[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:15 dist-test-slave-rdl9 krb5kdc[1695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:15 dist-test-slave-rdl9 krb5kdc[1695](info): set up 2 sockets
Jun 20 22:56:15 dist-test-slave-rdl9 krb5kdc[1695](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 22:56:16.734627 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:16 dist-test-slave-rdl9 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:16.852725 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46843
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:16.934223 1708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:16.934448 1708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:16.934468 1708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:16.935608 1708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:16.935648 1708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:16.935662 1708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:16.935685 1708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:16.935708 1708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:16.937148 1708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:46843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:16.937372 1708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:16.937594 1708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:56:16.942286 1708 server_base.cc:1034] running on GCE node
W20260620 22:56:16.942377 1713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:16.942451 1714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:16.942451 1716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:16.942850 1708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:16.943147 1708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:16.944293 1708 hybrid_clock.cc:648] HybridClock initialized: now 1781996176944274 us; error 33 us; skew 500 ppm
Jun 20 22:56:16 dist-test-slave-rdl9 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:16.946812 1708 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:16.947286 1708 webserver.cc:460] Webserver started at http://127.1.166.62:46425/ using document root <none> and password file <none>
I20260620 22:56:16.947500 1708 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:16.947543 1708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:16.947674 1708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:16.948745 1708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f"
format_stamp: "Formatted at 2026-06-20 22:56:16 on dist-test-slave-rdl9"
server_key: "d3c0d2088a24fb9e7b73ee370f2c730a"
server_key_iv: "dd3431ccc80d8fbf65f9de6e30446c37"
server_key_version: "encryptionkey@0"
I20260620 22:56:16.949095 1708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f"
format_stamp: "Formatted at 2026-06-20 22:56:16 on dist-test-slave-rdl9"
server_key: "d3c0d2088a24fb9e7b73ee370f2c730a"
server_key_iv: "dd3431ccc80d8fbf65f9de6e30446c37"
server_key_version: "encryptionkey@0"
I20260620 22:56:16.950474 1708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:16.951287 1723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:16.951462 1708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:16.951542 1708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f"
format_stamp: "Formatted at 2026-06-20 22:56:16 on dist-test-slave-rdl9"
server_key: "d3c0d2088a24fb9e7b73ee370f2c730a"
server_key_iv: "dd3431ccc80d8fbf65f9de6e30446c37"
server_key_version: "encryptionkey@0"
I20260620 22:56:16.951618 1708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:16.980185 1708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:16.982931 1708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:16.983119 1708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:16.988042 1708 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:46843
I20260620 22:56:16.988101 1775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:46843 every 8 connection(s)
I20260620 22:56:16.988469 1708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:16.989501 1776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:16.992152 1776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f: Bootstrap starting.
I20260620 22:56:16.992846 1776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:16.993146 1776 log.cc:824] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:16.994199 1776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f: No bootstrap required, opened a new log
I20260620 22:56:16.996351 1776 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46843 } }
I20260620 22:56:16.996583 1776 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:16.996611 1776 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ea5abf0c5d4e01bb8eddfcb020cc9f, State: Initialized, Role: FOLLOWER
I20260620 22:56:16.996798 1776 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46843 } }
I20260620 22:56:16.996886 1776 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:16.996914 1776 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:16.996959 1776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:16.997812 1776 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46843 } }
I20260620 22:56:16.997969 1776 leader_election.cc:302] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ea5abf0c5d4e01bb8eddfcb020cc9f; no voters:
I20260620 22:56:16.998188 1776 leader_election.cc:288] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:16.998229 1779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:16.998227 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 1708
I20260620 22:56:16.998369 1779 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [term 1 LEADER]: Becoming Leader. State: Replica: b3ea5abf0c5d4e01bb8eddfcb020cc9f, State: Running, Role: LEADER
I20260620 22:56:16.998390 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:16.998468 1779 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46843 } }
I20260620 22:56:16.998534 1776 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:16.998898 1780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46843 } } }
I20260620 22:56:16.999040 1780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:16.999284 1781 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3ea5abf0c5d4e01bb8eddfcb020cc9f. Latest consensus state: current_term: 1 leader_uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea5abf0c5d4e01bb8eddfcb020cc9f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46843 } } }
I20260620 22:56:16.999370 1781 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:16.999840 1783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:17.000023 1688 external_mini_cluster.cc:1428] Setting key f9eaf822a00ed1b45159c41d25065920
I20260620 22:56:17.000571 1783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:17.002447 1783 catalog_manager.cc:1347] Generated new cluster ID: 924a03d37664471484f2e28f17b83e1d
I20260620 22:56:17.002527 1783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:17.016968 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.001028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.1.166.62:46843)
0620 22:56:17.001940 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:17.001949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:17.001991 (+ 42us) server_negotiation.cc:407] Connection header received
0620 22:56:17.003730 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:17.003736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:17.003789 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:17.003911 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:17.005543 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.006633 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.007687 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.008102 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.008201 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.011627 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:17.011632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:17.011633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:17.011646 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:17.013853 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.014873 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.014878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.014878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.015047 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.015412 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.015413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.015413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.015862 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:17.016068 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:17.016464 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 22:56:17.016630 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":72,"threads_started":1}
I20260620 22:56:17.019044 1783 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:17.019611 1783 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:17.025084 1783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3ea5abf0c5d4e01bb8eddfcb020cc9f: Generated new TSK 0
I20260620 22:56:17.025314 1783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:17.051318 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46843
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:17.165172 1802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:17.165424 1802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:17.165467 1802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:17.166852 1802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:17.166908 1802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:17.166951 1802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:17.168608 1802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:17.168875 1802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:17.169104 1802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:17.173928 1808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:17.173931 1810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:17.174024 1807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:17.174305 1802 server_base.cc:1034] running on GCE node
I20260620 22:56:17.174561 1802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:17.174799 1802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:17.175969 1802 hybrid_clock.cc:648] HybridClock initialized: now 1781996177175951 us; error 28 us; skew 500 ppm
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:17.178526 1802 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:17.178998 1802 webserver.cc:460] Webserver started at http://127.1.166.1:43295/ using document root <none> and password file <none>
I20260620 22:56:17.179219 1802 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:17.179267 1802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:17.179400 1802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:17.180456 1802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "1862674f4eb343d68b1584a53d64959d"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "b56e0ca1724ff6bfca542545f15e66f0"
server_key_iv: "1748c544935dcd81514025647c2d4d0b"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.180805 1802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "1862674f4eb343d68b1584a53d64959d"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "b56e0ca1724ff6bfca542545f15e66f0"
server_key_iv: "1748c544935dcd81514025647c2d4d0b"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.182312 1802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:17.183251 1817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.183456 1802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:17.183547 1802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "1862674f4eb343d68b1584a53d64959d"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "b56e0ca1724ff6bfca542545f15e66f0"
server_key_iv: "1748c544935dcd81514025647c2d4d0b"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.183621 1802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:17.200162 1802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:17.202168 1802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:17.202306 1802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:17.202555 1802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:17.202862 1802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:17.202896 1802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.202925 1802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:17.202942 1802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.209656 1802 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:46259
I20260620 22:56:17.209695 1930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:46259 every 8 connection(s)
I20260620 22:56:17.210059 1802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:17.217862 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 1802
I20260620 22:56:17.217948 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:17.218086 1688 external_mini_cluster.cc:1428] Setting key 9f44268b5865dc95e07e0f6fdb744cda
I20260620 22:56:17.219858 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.210851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:44613 (local address 127.1.166.62:46843)
0620 22:56:17.210955 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:17.210957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:17.211093 (+ 136us) server_negotiation.cc:407] Connection header received
0620 22:56:17.211622 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:17.211623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:17.211641 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:17.211690 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:17.212853 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.213381 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.214578 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.214809 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.214868 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.217142 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:17.217144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:17.217144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:17.217151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:17.218597 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.218999 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.219000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.219000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.219062 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.219263 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.219264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.219264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.219393 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:17.219455 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:17.219679 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 22:56:17.219767 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 22:56:17.219969 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.210779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46843 (local address 127.1.166.1:44613)
0620 22:56:17.210964 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:17.211009 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:17.211517 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:17.211772 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:17.211775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:17.212491 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:17.212714 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:17.212720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.213473 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.213474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:17.214465 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:17.214468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.214885 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.214886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:17.215015 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.215564 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:17.215573 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:17.217007 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:17.218681 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:17.218689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:17.218691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:17.218933 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:17.219121 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:17.219121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:17.219122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:17.219215 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:17.219445 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:17.219449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:17.219567 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 22:56:17.219648 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 22:56:17.219783 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260620 22:56:17.220431 1931 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46843
I20260620 22:56:17.220527 1931 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:17.220702 1931 heartbeater.cc:511] Master 127.1.166.62:46843 requested a full tablet report, sending...
I20260620 22:56:17.221253 1740 ts_manager.cc:194] Registered new tserver with Master: 1862674f4eb343d68b1584a53d64959d (127.1.166.1:46259)
I20260620 22:56:17.222091 1740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:44613
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:17.242688 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46843
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:17.328437 1936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:17.328675 1936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:17.328702 1936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:17.330253 1936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:17.330331 1936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:17.330406 1936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:17.332114 1936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:17.332352 1936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:17.332551 1936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:17.337393 1944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:17.337393 1942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:17.337453 1941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:17.337623 1936 server_base.cc:1034] running on GCE node
I20260620 22:56:17.337832 1936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:17.338074 1936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:17.339299 1936 hybrid_clock.cc:648] HybridClock initialized: now 1781996177339257 us; error 69 us; skew 500 ppm
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:17.342461 1936 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:17.343209 1936 webserver.cc:460] Webserver started at http://127.1.166.2:33697/ using document root <none> and password file <none>
I20260620 22:56:17.343531 1936 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:17.343642 1936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:17.343844 1936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:17.345348 1936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "c6e543e0edd14f55a25a4c8c5e6f9cfa"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "dcd7f6078586931e44a4c3e8f9c498c2"
server_key_iv: "deb788fa46bd95299a456dd486a7bfe1"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.346004 1936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "c6e543e0edd14f55a25a4c8c5e6f9cfa"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "dcd7f6078586931e44a4c3e8f9c498c2"
server_key_iv: "deb788fa46bd95299a456dd486a7bfe1"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.348026 1936 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260620 22:56:17.348904 1951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.349192 1936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:17.349296 1936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "c6e543e0edd14f55a25a4c8c5e6f9cfa"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "dcd7f6078586931e44a4c3e8f9c498c2"
server_key_iv: "deb788fa46bd95299a456dd486a7bfe1"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.349649 1936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:17.362260 1936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:17.364565 1936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:17.364701 1936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:17.364951 1936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:17.365255 1936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:17.365288 1936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.365309 1936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:17.365341 1936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.372639 1936 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:41681
I20260620 22:56:17.372674 2064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:41681 every 8 connection(s)
I20260620 22:56:17.373039 1936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:17.378206 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 1936
I20260620 22:56:17.378360 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:17.378566 1688 external_mini_cluster.cc:1428] Setting key f6fddc2dafacb9346e8ee9c2d3eeb2e8
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:17.385622 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.373908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:39907 (local address 127.1.166.62:46843)
0620 22:56:17.373968 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:17.373970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:17.374769 (+ 799us) server_negotiation.cc:407] Connection header received
0620 22:56:17.375289 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:17.375291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:17.375303 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:17.375379 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:17.376518 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.377093 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.378865 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.379171 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.379243 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.381928 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:17.381930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:17.381930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:17.381938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:17.383549 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.384196 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.384198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.384199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.384271 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.384607 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.384609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.384609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.384767 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:17.384835 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:17.385431 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 22:56:17.385538 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:17.385964 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.373895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46843 (local address 127.1.166.2:39907)
0620 22:56:17.374102 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:17.374147 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:17.375136 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:17.375497 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:17.375501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:17.376135 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:17.376359 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:17.376364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.377245 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.377246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:17.378690 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:17.378697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.379323 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.379325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:17.379542 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.380099 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:17.380109 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:17.381753 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:17.383681 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:17.383695 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:17.383698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:17.384061 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:17.384348 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:17.384349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:17.384349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:17.384507 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:17.384844 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:17.384852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:17.385180 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 22:56:17.385368 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 22:56:17.385619 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
I20260620 22:56:17.386726 2065 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46843
I20260620 22:56:17.386858 2065 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:17.387250 2065 heartbeater.cc:511] Master 127.1.166.62:46843 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
I20260620 22:56:17.387809 1740 ts_manager.cc:194] Registered new tserver with Master: c6e543e0edd14f55a25a4c8c5e6f9cfa (127.1.166.2:41681)
I20260620 22:56:17.388342 1740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:39907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:17.403458 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46843
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:17.492568 2070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:17.492841 2070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:17.492871 2070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:17.494227 2070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:17.494292 2070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:17.494325 2070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:17.496194 2070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:17.496441 2070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:17.496655 2070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:17.502002 2075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:17.502002 2076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:17.502002 2078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:17.502223 2070 server_base.cc:1034] running on GCE node
I20260620 22:56:17.502506 2070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:17.502756 2070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:17.503942 2070 hybrid_clock.cc:648] HybridClock initialized: now 1781996177503923 us; error 29 us; skew 500 ppm
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:17.506678 2070 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:17.507190 2070 webserver.cc:460] Webserver started at http://127.1.166.3:42829/ using document root <none> and password file <none>
I20260620 22:56:17.507426 2070 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:17.507480 2070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:17.507587 2070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:17.508708 2070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "ab8e4c2ddfb94fc190ebd8bcdebc89a4"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "928d8021f85b065ef5505e487b9b928a"
server_key_iv: "573abbf20f0f47dee5be4b613d5a30e9"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.509114 2070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "ab8e4c2ddfb94fc190ebd8bcdebc89a4"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "928d8021f85b065ef5505e487b9b928a"
server_key_iv: "573abbf20f0f47dee5be4b613d5a30e9"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.510707 2070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 22:56:17.511838 2085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.512014 2070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:17.512185 2070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "ab8e4c2ddfb94fc190ebd8bcdebc89a4"
format_stamp: "Formatted at 2026-06-20 22:56:17 on dist-test-slave-rdl9"
server_key: "928d8021f85b065ef5505e487b9b928a"
server_key_iv: "573abbf20f0f47dee5be4b613d5a30e9"
server_key_version: "encryptionkey@0"
I20260620 22:56:17.512279 2070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:17.521590 2070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:17.524366 2070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:17.524530 2070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:17.524844 2070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:17.525391 2070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:17.525436 2070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.525462 2070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:17.525473 2070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:17.532526 2070 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:43707
I20260620 22:56:17.532624 2198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:43707 every 8 connection(s)
I20260620 22:56:17.532917 2070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:17.539439 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 2070
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:17.539520 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:17.539681 1688 external_mini_cluster.cc:1428] Setting key b8a7aa0bd2712c74df7a746251b1b8a0
I20260620 22:56:17.543021 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.533877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:34545 (local address 127.1.166.62:46843)
0620 22:56:17.533939 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:17.533941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:17.534133 (+ 192us) server_negotiation.cc:407] Connection header received
0620 22:56:17.534688 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:17.534689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:17.534699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:17.534736 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:17.535739 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.536258 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.537310 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.537531 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.537590 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.540029 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:17.540031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:17.540032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:17.540040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:17.541463 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.541946 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.541947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.541948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.542011 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.542285 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.542285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.542286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.542468 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:17.542536 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:17.542864 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 22:56:17.542944 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:56:17.543195 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.533834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46843 (local address 127.1.166.3:34545)
0620 22:56:17.534011 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:17.534054 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:17.534590 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:17.534800 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:17.534803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:17.535406 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:17.535613 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:17.535618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.536361 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.536371 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:17.537212 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:17.537214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.537632 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.537633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:17.537763 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.538305 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:17.538314 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:17.539910 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:17.541568 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:17.541578 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:17.541580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:17.541850 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:17.542100 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:17.542101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:17.542101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:17.542228 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:17.542557 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:17.542563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:17.542712 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 22:56:17.542817 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 22:56:17.542971 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260620 22:56:17.543812 2199 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46843
I20260620 22:56:17.543923 2199 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:17.544121 2199 heartbeater.cc:511] Master 127.1.166.62:46843 requested a full tablet report, sending...
I20260620 22:56:17.544595 1740 ts_manager.cc:194] Registered new tserver with Master: ab8e4c2ddfb94fc190ebd8bcdebc89a4 (127.1.166.3:43707)
I20260620 22:56:17.544975 1740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:34545
I20260620 22:56:17.552175 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:17.575124 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.567280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.1.166.1:46259)
0620 22:56:17.567564 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:17.567569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:17.567581 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:56:17.567607 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:17.567609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:17.567622 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:17.567694 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:17.568288 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.569019 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.569799 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.570123 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.570185 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.572509 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:17.572521 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:17.572521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:17.572529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:17.573882 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.574282 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.574285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.574286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.574394 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.574620 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.574622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.574622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.574792 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:17.574936 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:17.574972 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 22:56:17.575042 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":79,"threads_started":1}
W20260620 22:56:17.575481 1845 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51890
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:17.600989 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:17.592266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.1.166.1:46259)
0620 22:56:17.592339 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:17.592341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:17.592795 (+ 454us) server_negotiation.cc:407] Connection header received
0620 22:56:17.592884 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:17.592886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:17.592900 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:17.592958 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:17.593672 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.594304 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.595682 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:17.596011 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:17.596093 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:17.598478 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:17.598480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:17.598480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:17.598491 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:17.599872 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.600210 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.600211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.600212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.600271 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:17.600438 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:17.600439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:17.600439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:17.600551 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:17.600659 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:17.600804 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 22:56:17.600893 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:56:17.601536 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 1802
I20260620 22:56:17.606267 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 1936
I20260620 22:56:17.613135 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 2070
I20260620 22:56:17.617334 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 1708
2026-06-20T22:56:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1860 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[2213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[2213](info): set up 2 sockets
Jun 20 22:56:17 dist-test-slave-rdl9 krb5kdc[2213](info): commencing operation
krb5kdc: starting...
W20260620 22:56:18.589776 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:18 dist-test-slave-rdl9 krb5kdc[2213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:18.700666 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:43489
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:43489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:18.781349 2226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:18.781585 2226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:18.781610 2226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:18.782928 2226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:18.782986 2226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:18.783000 2226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:18.783012 2226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:18.783023 2226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:18.784667 2226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:43489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:43489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:18.784917 2226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:18.785152 2226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:18.789716 2232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:18.789786 2231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:18.789726 2234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:18.789950 2226 server_base.cc:1034] running on GCE node
I20260620 22:56:18.790218 2226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:18.790548 2226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:18.791718 2226 hybrid_clock.cc:648] HybridClock initialized: now 1781996178791691 us; error 39 us; skew 500 ppm
Jun 20 22:56:18 dist-test-slave-rdl9 krb5kdc[2213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996178, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:18.794498 2226 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:18.794986 2226 webserver.cc:460] Webserver started at http://127.1.166.62:43537/ using document root <none> and password file <none>
I20260620 22:56:18.795220 2226 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:18.795274 2226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:18.795410 2226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:18.796485 2226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "cf2602f838954e17b4b17666e29c444e"
format_stamp: "Formatted at 2026-06-20 22:56:18 on dist-test-slave-rdl9"
server_key: "0c2595c8331d31f1f7c4cc3f2d01c92d"
server_key_iv: "457be793938c84efe8d85488e72aa459"
server_key_version: "encryptionkey@0"
I20260620 22:56:18.796844 2226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "cf2602f838954e17b4b17666e29c444e"
format_stamp: "Formatted at 2026-06-20 22:56:18 on dist-test-slave-rdl9"
server_key: "0c2595c8331d31f1f7c4cc3f2d01c92d"
server_key_iv: "457be793938c84efe8d85488e72aa459"
server_key_version: "encryptionkey@0"
I20260620 22:56:18.798286 2226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:56:18.799153 2241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:18.799347 2226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:18.799484 2226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "cf2602f838954e17b4b17666e29c444e"
format_stamp: "Formatted at 2026-06-20 22:56:18 on dist-test-slave-rdl9"
server_key: "0c2595c8331d31f1f7c4cc3f2d01c92d"
server_key_iv: "457be793938c84efe8d85488e72aa459"
server_key_version: "encryptionkey@0"
I20260620 22:56:18.799572 2226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:18.815924 2226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:18.818753 2226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:18.818917 2226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:18.822603 2226 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:43489
I20260620 22:56:18.822654 2293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:43489 every 8 connection(s)
I20260620 22:56:18.823033 2226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:18.824021 2294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:18.825873 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 2226
I20260620 22:56:18.825999 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:18.826179 1688 external_mini_cluster.cc:1428] Setting key 260fbfe219371bdbddeee615072be307
I20260620 22:56:18.826844 2294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e: Bootstrap starting.
I20260620 22:56:18.827603 2294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:18.827940 2294 log.cc:824] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:18.828634 2294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e: No bootstrap required, opened a new log
I20260620 22:56:18.830940 2294 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2602f838954e17b4b17666e29c444e" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43489 } }
I20260620 22:56:18.831126 2294 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:18.831159 2294 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2602f838954e17b4b17666e29c444e, State: Initialized, Role: FOLLOWER
I20260620 22:56:18.831287 2294 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2602f838954e17b4b17666e29c444e" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43489 } }
I20260620 22:56:18.831362 2294 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:18.831395 2294 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:18.831439 2294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:18.832096 2294 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2602f838954e17b4b17666e29c444e" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43489 } }
I20260620 22:56:18.832201 2294 leader_election.cc:302] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf2602f838954e17b4b17666e29c444e; no voters:
I20260620 22:56:18.832360 2294 leader_election.cc:288] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:18.832420 2299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:18.832576 2299 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [term 1 LEADER]: Becoming Leader. State: Replica: cf2602f838954e17b4b17666e29c444e, State: Running, Role: LEADER
I20260620 22:56:18.832615 2294 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:18.832675 2299 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2602f838954e17b4b17666e29c444e" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43489 } }
I20260620 22:56:18.833076 2301 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2602f838954e17b4b17666e29c444e. Latest consensus state: current_term: 1 leader_uuid: "cf2602f838954e17b4b17666e29c444e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2602f838954e17b4b17666e29c444e" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43489 } } }
I20260620 22:56:18.833093 2300 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf2602f838954e17b4b17666e29c444e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2602f838954e17b4b17666e29c444e" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43489 } } }
I20260620 22:56:18.833180 2301 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:18.833190 2300 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:18.833387 2308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 22:56:18 dist-test-slave-rdl9 krb5kdc[2213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:18.833884 2308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:18.835795 2308 catalog_manager.cc:1347] Generated new cluster ID: 0d7d891bb502430ba6e0eba27b2fbe92
I20260620 22:56:18.835871 2308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:18.838181 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:18.826916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.1.166.62:43489)
0620 22:56:18.827097 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:18.827106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:18.827136 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:56:18.827743 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:18.827754 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:18.827825 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:18.827987 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:18.828694 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:18.830062 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:18.831004 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:18.831610 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:18.831724 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:18.834385 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:18.834391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:18.834391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:18.834405 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:18.836429 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:18.836851 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:18.836855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:18.836855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:18.836962 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:18.837162 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:18.837163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:18.837163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:18.837494 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:18.837641 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:18.837762 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 22:56:18.837913 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260620 22:56:18.848400 2308 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:18.849318 2308 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:18.860211 2308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf2602f838954e17b4b17666e29c444e: Generated new TSK 0
I20260620 22:56:18.860448 2308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:18.890537 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43489
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:18.975449 2320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:18.975682 2320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:18.975708 2320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:18.976972 2320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:18.977031 2320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:18.977061 2320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:18.978715 2320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:18.978960 2320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:18.979158 2320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:18.984227 2328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:18.984236 2326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:18.984241 2325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:18.984476 2320 server_base.cc:1034] running on GCE node
I20260620 22:56:18.984714 2320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:18.984992 2320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:18.986161 2320 hybrid_clock.cc:648] HybridClock initialized: now 1781996178986144 us; error 29 us; skew 500 ppm
Jun 20 22:56:18 dist-test-slave-rdl9 krb5kdc[2213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996178, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:18.989377 2320 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:18.989861 2320 webserver.cc:460] Webserver started at http://127.1.166.1:38397/ using document root <none> and password file <none>
I20260620 22:56:18.990067 2320 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:18.990115 2320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:18.990240 2320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:18.991389 2320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "0ebcf345839a4182926230761d68310c"
format_stamp: "Formatted at 2026-06-20 22:56:18 on dist-test-slave-rdl9"
server_key: "be63519e8b0aad9ea55ab289fa478980"
server_key_iv: "b87cb4a602437c77b255cf4c020cb5b7"
server_key_version: "encryptionkey@0"
I20260620 22:56:18.991750 2320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "0ebcf345839a4182926230761d68310c"
format_stamp: "Formatted at 2026-06-20 22:56:18 on dist-test-slave-rdl9"
server_key: "be63519e8b0aad9ea55ab289fa478980"
server_key_iv: "b87cb4a602437c77b255cf4c020cb5b7"
server_key_version: "encryptionkey@0"
I20260620 22:56:18.993103 2320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:18.993816 2335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:18.994012 2320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:18.994102 2320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "0ebcf345839a4182926230761d68310c"
format_stamp: "Formatted at 2026-06-20 22:56:18 on dist-test-slave-rdl9"
server_key: "be63519e8b0aad9ea55ab289fa478980"
server_key_iv: "b87cb4a602437c77b255cf4c020cb5b7"
server_key_version: "encryptionkey@0"
I20260620 22:56:18.994179 2320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:19.027211 2320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:19.029599 2320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:19.029738 2320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:19.029973 2320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:19.030273 2320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:19.030308 2320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:19.030372 2320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:19.030398 2320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:19.036252 2320 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:43003
I20260620 22:56:19.036317 2448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:43003 every 8 connection(s)
I20260620 22:56:19.036638 2320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996178, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:19.046021 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 2320
I20260620 22:56:19.046126 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:19.046301 1688 external_mini_cluster.cc:1428] Setting key 94497bb4a12087b48f7098a3d06da3aa
I20260620 22:56:19.047726 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.037461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:38009 (local address 127.1.166.62:43489)
0620 22:56:19.037550 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.037553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.037771 (+ 218us) server_negotiation.cc:407] Connection header received
0620 22:56:19.038821 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.038823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.038836 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.038908 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:19.039816 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.040356 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.041740 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.042004 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.042073 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.044679 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:19.044681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:19.044682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:19.044692 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:19.046173 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.046648 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.046649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.046650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.046709 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.047017 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.047018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.047018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.047175 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:19.047247 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.047520 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.047631 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 22:56:19.047792 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.037371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:43489 (local address 127.1.166.1:38009)
0620 22:56:19.037636 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:19.037676 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:19.038676 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:19.038982 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:19.038985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:19.039283 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:19.039621 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.039635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.040524 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.040525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.041590 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.041593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.042166 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.042168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.042416 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.043025 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:19.043035 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:19.044531 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:19.046275 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:19.046285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:19.046287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:19.046585 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:19.046765 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:19.046766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:19.046766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:19.046918 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:19.047236 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:19.047240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:19.047383 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 22:56:19.047467 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:56:19.047610 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260620 22:56:19.048273 2449 heartbeater.cc:347] Connected to a master server at 127.1.166.62:43489
I20260620 22:56:19.048385 2449 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:19.048592 2449 heartbeater.cc:511] Master 127.1.166.62:43489 requested a full tablet report, sending...
I20260620 22:56:19.049108 2258 ts_manager.cc:194] Registered new tserver with Master: 0ebcf345839a4182926230761d68310c (127.1.166.1:43003)
I20260620 22:56:19.049963 2258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:38009
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:19.071452 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43489
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:19.159363 2454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:19.159590 2454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:19.159617 2454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:19.160964 2454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:19.161031 2454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:19.161063 2454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:19.162797 2454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:19.163069 2454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:19.163319 2454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:19.168478 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:19.168484 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:19.168478 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:19.168794 2454 server_base.cc:1034] running on GCE node
I20260620 22:56:19.168993 2454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:19.169225 2454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:19.170428 2454 hybrid_clock.cc:648] HybridClock initialized: now 1781996179170337 us; error 100 us; skew 500 ppm
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:19.173005 2454 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:19.173468 2454 webserver.cc:460] Webserver started at http://127.1.166.2:39731/ using document root <none> and password file <none>
I20260620 22:56:19.173693 2454 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:19.173743 2454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:19.173872 2454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:19.174979 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97"
format_stamp: "Formatted at 2026-06-20 22:56:19 on dist-test-slave-rdl9"
server_key: "39d90eb7ab368bbdf2b943724306cee3"
server_key_iv: "fc7d854e7fb6dd65e96e3e63c627599a"
server_key_version: "encryptionkey@0"
I20260620 22:56:19.175432 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97"
format_stamp: "Formatted at 2026-06-20 22:56:19 on dist-test-slave-rdl9"
server_key: "39d90eb7ab368bbdf2b943724306cee3"
server_key_iv: "fc7d854e7fb6dd65e96e3e63c627599a"
server_key_version: "encryptionkey@0"
I20260620 22:56:19.176805 2454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:19.177786 2469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:19.178020 2454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:19.178105 2454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97"
format_stamp: "Formatted at 2026-06-20 22:56:19 on dist-test-slave-rdl9"
server_key: "39d90eb7ab368bbdf2b943724306cee3"
server_key_iv: "fc7d854e7fb6dd65e96e3e63c627599a"
server_key_version: "encryptionkey@0"
I20260620 22:56:19.178176 2454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:19.194728 2454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:19.197237 2454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:19.197395 2454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:19.197635 2454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:19.197964 2454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:19.198001 2454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:19.198031 2454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:19.198055 2454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:19.204519 2454 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:35365
I20260620 22:56:19.204639 2582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:35365 every 8 connection(s)
I20260620 22:56:19.204934 2454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:19.207175 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 2454
I20260620 22:56:19.207288 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:19.207556 1688 external_mini_cluster.cc:1428] Setting key 13f3249d811ca197d8936958692ce4c9
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
I20260620 22:56:19.217871 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.205861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:52167 (local address 127.1.166.62:43489)
0620 22:56:19.205946 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.205949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.206139 (+ 190us) server_negotiation.cc:407] Connection header received
0620 22:56:19.207221 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.207224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.207238 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.207291 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:19.208117 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.209103 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.210943 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.211220 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.211297 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.214694 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:19.214696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:19.214696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:19.214706 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:19.216125 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.216737 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.216738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.216738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.216801 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.217142 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.217143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.217143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.217291 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:19.217368 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.217710 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.217787 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 22:56:19.218024 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.205817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:43489 (local address 127.1.166.2:52167)
0620 22:56:19.206004 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:19.206046 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:19.207098 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:19.207386 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:19.207391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:19.207685 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:19.208010 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.208016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.209596 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.209597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.210626 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.210629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.211346 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.211348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.211620 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.212277 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:19.212289 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:19.214544 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:19.216234 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:19.216245 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:19.216248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:19.216617 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:19.216874 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:19.216874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:19.216875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:19.217031 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:19.217372 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:19.217379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:19.217570 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 22:56:19.217675 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 22:56:19.217822 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260620 22:56:19.218568 2583 heartbeater.cc:347] Connected to a master server at 127.1.166.62:43489
I20260620 22:56:19.218681 2583 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:19.218894 2583 heartbeater.cc:511] Master 127.1.166.62:43489 requested a full tablet report, sending...
I20260620 22:56:19.219322 2258 ts_manager.cc:194] Registered new tserver with Master: 8c538c9a9a6b4fa1ad5b5c9ea7063a97 (127.1.166.2:35365)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
I20260620 22:56:19.219739 2258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:52167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:19.233486 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43489
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:19.314667 2588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:19.315006 2588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:19.315053 2588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:19.316381 2588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:19.316432 2588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:19.316464 2588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:19.318277 2588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:19.318579 2588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:19.318801 2588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:19.323696 2593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:19.323696 2596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:19.323696 2594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:19.324030 2588 server_base.cc:1034] running on GCE node
I20260620 22:56:19.324259 2588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:19.324600 2588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:19.325809 2588 hybrid_clock.cc:648] HybridClock initialized: now 1781996179325796 us; error 37 us; skew 500 ppm
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:19.328785 2588 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:19.329232 2588 webserver.cc:460] Webserver started at http://127.1.166.3:41627/ using document root <none> and password file <none>
I20260620 22:56:19.329421 2588 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:19.329463 2588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:19.329566 2588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:19.330704 2588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "49062d885fe54b05ba136429ac730498"
format_stamp: "Formatted at 2026-06-20 22:56:19 on dist-test-slave-rdl9"
server_key: "9a29f70b863db50c24615a9d0ee384b5"
server_key_iv: "1a53948fe41ac9b0189615e200d11edf"
server_key_version: "encryptionkey@0"
I20260620 22:56:19.331017 2588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "49062d885fe54b05ba136429ac730498"
format_stamp: "Formatted at 2026-06-20 22:56:19 on dist-test-slave-rdl9"
server_key: "9a29f70b863db50c24615a9d0ee384b5"
server_key_iv: "1a53948fe41ac9b0189615e200d11edf"
server_key_version: "encryptionkey@0"
I20260620 22:56:19.332360 2588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:19.333762 2603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:19.334007 2588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:19.334123 2588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "49062d885fe54b05ba136429ac730498"
format_stamp: "Formatted at 2026-06-20 22:56:19 on dist-test-slave-rdl9"
server_key: "9a29f70b863db50c24615a9d0ee384b5"
server_key_iv: "1a53948fe41ac9b0189615e200d11edf"
server_key_version: "encryptionkey@0"
I20260620 22:56:19.334220 2588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:19.357105 2588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:19.359478 2588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:19.359650 2588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:19.359910 2588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:19.360235 2588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:19.360294 2588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:19.360325 2588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:19.360342 2588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:19.366689 2588 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:38687
I20260620 22:56:19.366734 2716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:38687 every 8 connection(s)
I20260620 22:56:19.367084 2588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:19.369014 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 2588
I20260620 22:56:19.369109 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:19.369267 1688 external_mini_cluster.cc:1428] Setting key b003dd21ac179f260e4b70b724c9ae9f
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996179, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:19.377337 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.367841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:38635 (local address 127.1.166.62:43489)
0620 22:56:19.367886 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.367889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.368109 (+ 220us) server_negotiation.cc:407] Connection header received
0620 22:56:19.369183 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.369184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.369198 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.369253 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:19.369974 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.370535 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.371596 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.371798 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.371854 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.374524 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:19.374526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:19.374526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:19.374536 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:19.375924 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.376381 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.376382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.376383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.376438 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.376665 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.376666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.376667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.376810 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:19.376868 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.377146 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.377230 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:56:19.377527 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.367817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:43489 (local address 127.1.166.3:38635)
0620 22:56:19.367983 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:19.368022 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:19.368996 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:19.369356 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:19.369361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:19.369660 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:19.369876 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.369881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.370634 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.370635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.371504 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.371507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.371871 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.371872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.372004 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.372621 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:19.372635 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:19.374414 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:19.376038 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:19.376049 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:19.376052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:19.376308 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:19.376497 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:19.376498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:19.376499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:19.376609 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:19.376867 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:19.376874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:19.377025 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 22:56:19.377119 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:56:19.377294 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260620 22:56:19.378191 2717 heartbeater.cc:347] Connected to a master server at 127.1.166.62:43489
I20260620 22:56:19.378286 2717 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:19.378515 2717 heartbeater.cc:511] Master 127.1.166.62:43489 requested a full tablet report, sending...
I20260620 22:56:19.379032 2258 ts_manager.cc:194] Registered new tserver with Master: 49062d885fe54b05ba136429ac730498 (127.1.166.3:38687)
I20260620 22:56:19.379487 2258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:38635
I20260620 22:56:19.380903 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:56:19.389165 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.382488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.1.166.62:43489)
0620 22:56:19.382535 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.382537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.382689 (+ 152us) server_negotiation.cc:407] Connection header received
0620 22:56:19.382717 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.382717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.382728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.382773 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:19.383295 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.383797 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.384746 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.384951 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.385023 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.386279 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:19.386280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:19.386281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:19.386289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:19.387616 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.387994 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.387995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.387996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.388058 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.388423 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.388424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.388425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.388602 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:19.388702 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.388949 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.389075 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:56:19.392690 2258 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:19.395200 2258 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:19.402184 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.398229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.1.166.1:43003)
0620 22:56:19.398511 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.398516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.398527 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:19.398591 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.398594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.398612 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.398688 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:19.398870 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.399958 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.401108 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.401671 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.401751 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.401814 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.402049 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.402096 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":176,"threads_started":1}
I20260620 22:56:19.402202 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.397892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:43003 (local address 127.0.0.1:33158)
0620 22:56:19.398452 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:19.398463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:19.398520 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:19.398702 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:19.398706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:19.398708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:19.398805 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.398808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.400047 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.400048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.401021 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.401023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.401850 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.401851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.401934 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.401947 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:56:19.402018 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:56:19.402073 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":1,"thread_start_us":139,"threads_started":1}
I20260620 22:56:19.402601 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.397825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:38687 (local address 127.0.0.1:49982)
0620 22:56:19.398012 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:19.398051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:19.398119 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:19.399114 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:19.399115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:19.399119 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:19.399184 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.399186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.400241 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.400242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.401470 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.401472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.402414 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.402415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.402471 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.402475 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:19.402501 (+ 26us) client_negotiation.cc:240] Negotiation successful
0620 22:56:19.402537 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260620 22:56:19.402827 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.397825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:35365 (local address 127.0.0.1:43846)
0620 22:56:19.398212 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:19.398225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:19.398306 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:19.398716 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:19.398717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:19.398720 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:19.398837 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.398841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.400285 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.400286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.401470 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.401472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.402523 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.402524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.402571 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.402575 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:19.402619 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:56:19.402754 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":27,"thread_start_us":71,"threads_started":1}
I20260620 22:56:19.402844 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.397994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.1.166.2:35365)
0620 22:56:19.398530 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.398535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.398545 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:19.398587 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.398589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.398604 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.398687 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:19.399025 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.400185 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.401568 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.402368 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.402435 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.402490 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.402726 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.402769 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":92,"threads_started":1}
I20260620 22:56:19.402958 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.398094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.1.166.3:38687)
0620 22:56:19.398925 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.398931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.398946 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:56:19.399001 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.399003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.399022 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.399105 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:19.399256 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.400142 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.401554 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.402297 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.402712 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.402782 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.402806 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.402847 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":97,"threads_started":1}
I20260620 22:56:19.403337 2383 tablet_service.cc:1467] Processing CreateTablet for tablet ad08bc0dd2b5440f9c63737a325e4352 (DEFAULT_TABLE table=test-table [id=89602b9ab32a48a8b4921dab5f29bab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:19.403580 2651 tablet_service.cc:1467] Processing CreateTablet for tablet ad08bc0dd2b5440f9c63737a325e4352 (DEFAULT_TABLE table=test-table [id=89602b9ab32a48a8b4921dab5f29bab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:19.403622 2517 tablet_service.cc:1467] Processing CreateTablet for tablet ad08bc0dd2b5440f9c63737a325e4352 (DEFAULT_TABLE table=test-table [id=89602b9ab32a48a8b4921dab5f29bab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:19.403724 2383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad08bc0dd2b5440f9c63737a325e4352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:19.403899 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad08bc0dd2b5440f9c63737a325e4352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:19.403899 2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad08bc0dd2b5440f9c63737a325e4352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:19.407390 2734 tablet_bootstrap.cc:492] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c: Bootstrap starting.
I20260620 22:56:19.408165 2734 tablet_bootstrap.cc:654] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:19.408671 2735 tablet_bootstrap.cc:492] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97: Bootstrap starting.
I20260620 22:56:19.408967 2734 log.cc:824] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:19.409135 2736 tablet_bootstrap.cc:492] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498: Bootstrap starting.
I20260620 22:56:19.409365 2735 tablet_bootstrap.cc:654] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:19.409927 2735 log.cc:824] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:19.410029 2736 tablet_bootstrap.cc:654] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:19.410691 2734 tablet_bootstrap.cc:492] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c: No bootstrap required, opened a new log
I20260620 22:56:19.410785 2734 ts_tablet_manager.cc:1397] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:56:19.410771 2736 log.cc:824] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:19.410838 2735 tablet_bootstrap.cc:492] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97: No bootstrap required, opened a new log
I20260620 22:56:19.410903 2735 ts_tablet_manager.cc:1397] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:56:19.411561 2736 tablet_bootstrap.cc:492] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498: No bootstrap required, opened a new log
I20260620 22:56:19.411638 2736 ts_tablet_manager.cc:1397] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:56:19.412753 2734 raft_consensus.cc:348] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.412978 2734 raft_consensus.cc:374] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:19.413018 2734 raft_consensus.cc:729] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebcf345839a4182926230761d68310c, State: Initialized, Role: FOLLOWER
I20260620 22:56:19.413184 2734 consensus_queue.cc:260] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.413039 2735 raft_consensus.cc:348] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.413273 2735 raft_consensus.cc:374] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:19.413327 2735 raft_consensus.cc:729] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c538c9a9a6b4fa1ad5b5c9ea7063a97, State: Initialized, Role: FOLLOWER
I20260620 22:56:19.413487 2735 consensus_queue.cc:260] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.413551 2449 heartbeater.cc:503] Master 127.1.166.62:43489 was elected leader, sending a full tablet report...
I20260620 22:56:19.413607 2734 ts_tablet_manager.cc:1428] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:56:19.413826 2735 ts_tablet_manager.cc:1428] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:56:19.413695 2736 raft_consensus.cc:348] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.413872 2736 raft_consensus.cc:374] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:19.413902 2736 raft_consensus.cc:729] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49062d885fe54b05ba136429ac730498, State: Initialized, Role: FOLLOWER
I20260620 22:56:19.413992 2583 heartbeater.cc:503] Master 127.1.166.62:43489 was elected leader, sending a full tablet report...
I20260620 22:56:19.414006 2736 consensus_queue.cc:260] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.414264 2736 ts_tablet_manager.cc:1428] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:56:19.414408 2717 heartbeater.cc:503] Master 127.1.166.62:43489 was elected leader, sending a full tablet report...
W20260620 22:56:19.455588 2584 tablet.cc:2369] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:56:19.537410 2450 tablet.cc:2369] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:56:19.617723 2718 tablet.cc:2369] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:19.648458 2742 raft_consensus.cc:482] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:19.648597 2742 raft_consensus.cc:504] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.649027 2742 leader_election.cc:288] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ebcf345839a4182926230761d68310c (127.1.166.1:43003), 8c538c9a9a6b4fa1ad5b5c9ea7063a97 (127.1.166.2:35365)
I20260620 22:56:19.654186 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.649282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:58459 (local address 127.1.166.2:35365)
0620 22:56:19.649331 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.649333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.649677 (+ 344us) server_negotiation.cc:407] Connection header received
0620 22:56:19.649714 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.649715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.649729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.649788 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:19.650087 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.651062 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.653006 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.653641 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.653747 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.653758 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.654062 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.654115 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:56:19.654186 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.649278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:46971 (local address 127.1.166.1:43003)
0620 22:56:19.649321 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.649323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.649700 (+ 377us) server_negotiation.cc:407] Connection header received
0620 22:56:19.649795 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.649796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.649814 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.649863 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:19.650087 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.651059 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.653006 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.653641 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.653747 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.653758 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.654063 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.654111 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:56:19.654191 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.649191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:35365 (local address 127.1.166.3:58459)
0620 22:56:19.649252 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:19.649264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:19.649320 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:19.649825 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:19.649826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:19.649828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:19.649903 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.649905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.651199 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.651200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.652814 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.652822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.653808 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.653810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.653924 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.653931 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 22:56:19.654016 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:56:19.654084 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":161}
I20260620 22:56:19.654196 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.649201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:43003 (local address 127.1.166.3:46971)
0620 22:56:19.649412 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:19.649419 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:19.649722 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:19.649877 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:19.649878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:19.649880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:19.649934 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.649935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.651199 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.651205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.652814 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:19.652822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.653808 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.653810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:19.653924 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.653932 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 22:56:19.654016 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:56:19.654082 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":109,"thread_start_us":94,"threads_started":1}
I20260620 22:56:19.654623 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad08bc0dd2b5440f9c63737a325e4352" candidate_uuid: "49062d885fe54b05ba136429ac730498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" is_pre_election: true
I20260620 22:56:19.654628 2403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad08bc0dd2b5440f9c63737a325e4352" candidate_uuid: "49062d885fe54b05ba136429ac730498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebcf345839a4182926230761d68310c" is_pre_election: true
I20260620 22:56:19.654789 2537 raft_consensus.cc:2454] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49062d885fe54b05ba136429ac730498 in term 0.
I20260620 22:56:19.654788 2403 raft_consensus.cc:2454] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49062d885fe54b05ba136429ac730498 in term 0.
I20260620 22:56:19.655055 2607 leader_election.cc:302] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebcf345839a4182926230761d68310c, 49062d885fe54b05ba136429ac730498; no voters:
I20260620 22:56:19.655196 2742 raft_consensus.cc:2781] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:19.655253 2742 raft_consensus.cc:482] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:19.655268 2742 raft_consensus.cc:3037] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:19.656009 2742 raft_consensus.cc:504] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.656149 2742 leader_election.cc:288] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [CANDIDATE]: Term 1 election: Requested vote from peers 0ebcf345839a4182926230761d68310c (127.1.166.1:43003), 8c538c9a9a6b4fa1ad5b5c9ea7063a97 (127.1.166.2:35365)
I20260620 22:56:19.656359 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad08bc0dd2b5440f9c63737a325e4352" candidate_uuid: "49062d885fe54b05ba136429ac730498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97"
I20260620 22:56:19.656360 2403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad08bc0dd2b5440f9c63737a325e4352" candidate_uuid: "49062d885fe54b05ba136429ac730498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebcf345839a4182926230761d68310c"
I20260620 22:56:19.656435 2537 raft_consensus.cc:3037] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:19.656437 2403 raft_consensus.cc:3037] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:19.657143 2537 raft_consensus.cc:2454] T ad08bc0dd2b5440f9c63737a325e4352 P 8c538c9a9a6b4fa1ad5b5c9ea7063a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49062d885fe54b05ba136429ac730498 in term 1.
I20260620 22:56:19.657192 2403 raft_consensus.cc:2454] T ad08bc0dd2b5440f9c63737a325e4352 P 0ebcf345839a4182926230761d68310c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49062d885fe54b05ba136429ac730498 in term 1.
I20260620 22:56:19.657326 2604 leader_election.cc:302] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49062d885fe54b05ba136429ac730498, 8c538c9a9a6b4fa1ad5b5c9ea7063a97; no voters:
I20260620 22:56:19.657410 2742 raft_consensus.cc:2781] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:19.657606 2742 raft_consensus.cc:686] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [term 1 LEADER]: Becoming Leader. State: Replica: 49062d885fe54b05ba136429ac730498, State: Running, Role: LEADER
I20260620 22:56:19.657708 2742 consensus_queue.cc:237] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } }
I20260620 22:56:19.658602 2258 catalog_manager.cc:5495] T ad08bc0dd2b5440f9c63737a325e4352 P 49062d885fe54b05ba136429ac730498 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49062d885fe54b05ba136429ac730498 (127.1.166.3). New cstate: current_term: 1 leader_uuid: "49062d885fe54b05ba136429ac730498" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebcf345839a4182926230761d68310c" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49062d885fe54b05ba136429ac730498" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c538c9a9a6b4fa1ad5b5c9ea7063a97" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 35365 } health_report { overall_health: UNKNOWN } } }
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:19.700531 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.688038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.1.166.1:43003)
0620 22:56:19.688137 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.688140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.688263 (+ 123us) server_negotiation.cc:407] Connection header received
0620 22:56:19.688329 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.688330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.688342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.688396 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:19.689055 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.689769 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.690644 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.690835 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.690892 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.697798 (+ 6906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:19.697812 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:19.697813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:19.697821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:19.699397 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.699757 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.699761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.699761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.699824 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.699987 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.699988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.699989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.700144 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:19.700288 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.700349 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.700435 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:19.726037 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.718719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.1.166.1:43003)
0620 22:56:19.718784 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.718786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.718802 (+ 16us) server_negotiation.cc:407] Connection header received
0620 22:56:19.718865 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.718866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.718876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.718925 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:19.719573 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.720044 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.720820 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.721014 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.721078 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.723401 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:19.723402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:19.723402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:19.723410 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:19.724651 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.725074 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.725075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.725075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.725134 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.725388 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.725389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.725390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.725506 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:19.725607 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.725880 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.725959 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:19.751133 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:19.742586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.1.166.1:43003)
0620 22:56:19.742701 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:19.742704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:19.742811 (+ 107us) server_negotiation.cc:407] Connection header received
0620 22:56:19.742891 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:19.742892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:19.742903 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:19.742948 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:19.743598 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.744078 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.744811 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:19.745008 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:19.745293 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:19.748258 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:19.748260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:19.748260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:19.748270 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:19.749712 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.750365 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.750367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.750368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.750445 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:19.750629 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:19.750629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:19.750630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:19.750739 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:19.750830 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:19.750946 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 22:56:19.751035 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 22:56:19.751655 2363 tablet_service.cc:2908] Rejecting scan request for tablet ad08bc0dd2b5440f9c63737a325e4352: Uninitialized: clean time has not yet been initialized
I20260620 22:56:19.752534 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 2320
I20260620 22:56:19.757309 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 2454
I20260620 22:56:19.761479 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 2588
I20260620 22:56:19.765723 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 2226
2026-06-20T22:56:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2151 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2758](info): set up 2 sockets
Jun 20 22:56:19 dist-test-slave-rdl9 krb5kdc[2758](info): commencing operation
krb5kdc: starting...
W20260620 22:56:20.757689 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:20 dist-test-slave-rdl9 krb5kdc[2758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:20.873216 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34951
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:34951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:20.957042 2771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:20.957304 2771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:20.957340 2771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:20.958770 2771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:20.958832 2771 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:56:20.958849 2771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:20.958866 2771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:20.958882 2771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:20.958896 2771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:20.960570 2771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:34951
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:20.960857 2771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:20.961076 2771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:20.966200 2776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:20.966216 2779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:20.966198 2777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:20.966506 2771 server_base.cc:1034] running on GCE node
I20260620 22:56:20.966810 2771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:20.967096 2771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:20.968271 2771 hybrid_clock.cc:648] HybridClock initialized: now 1781996180968256 us; error 32 us; skew 500 ppm
Jun 20 22:56:20 dist-test-slave-rdl9 krb5kdc[2758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:20.971433 2771 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:20.971913 2771 webserver.cc:460] Webserver started at http://127.1.166.62:42249/ using document root <none> and password file <none>
I20260620 22:56:20.972121 2771 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:20.972179 2771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:20.972304 2771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:20.973410 2771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "d20b3efb3f134cf99ff144202088cbc5"
format_stamp: "Formatted at 2026-06-20 22:56:20 on dist-test-slave-rdl9"
server_key: "67a4ded63b02d67ccfc9fdcf126bfe78"
server_key_iv: "521992ff1a45e6259f448bbc6b6e9de6"
server_key_version: "encryptionkey@0"
I20260620 22:56:20.973740 2771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "d20b3efb3f134cf99ff144202088cbc5"
format_stamp: "Formatted at 2026-06-20 22:56:20 on dist-test-slave-rdl9"
server_key: "67a4ded63b02d67ccfc9fdcf126bfe78"
server_key_iv: "521992ff1a45e6259f448bbc6b6e9de6"
server_key_version: "encryptionkey@0"
I20260620 22:56:20.975111 2771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:56:20.975961 2786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:20.976269 2771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:20.976415 2771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "d20b3efb3f134cf99ff144202088cbc5"
format_stamp: "Formatted at 2026-06-20 22:56:20 on dist-test-slave-rdl9"
server_key: "67a4ded63b02d67ccfc9fdcf126bfe78"
server_key_iv: "521992ff1a45e6259f448bbc6b6e9de6"
server_key_version: "encryptionkey@0"
I20260620 22:56:20.976516 2771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:20.987452 2771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:20.989702 2771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:20.989846 2771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:20.994463 2771 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:34951
I20260620 22:56:20.994521 2848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:34951 every 8 connection(s)
I20260620 22:56:20.994849 2771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:20.995577 2849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:20.998696 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 2771
I20260620 22:56:20.998787 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:20.998782 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5: Bootstrap starting.
I20260620 22:56:20.998941 1688 external_mini_cluster.cc:1428] Setting key 4d8ef4fc1128fc56e5e3d7e53841d452
I20260620 22:56:20.999588 2849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:20.999974 2849 log.cc:824] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:21.000926 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5: No bootstrap required, opened a new log
I20260620 22:56:21.003558 2849 raft_consensus.cc:348] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20b3efb3f134cf99ff144202088cbc5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34951 } }
I20260620 22:56:21.003813 2849 raft_consensus.cc:374] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:21.003847 2849 raft_consensus.cc:729] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20b3efb3f134cf99ff144202088cbc5, State: Initialized, Role: FOLLOWER
I20260620 22:56:21.003988 2849 consensus_queue.cc:260] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20b3efb3f134cf99ff144202088cbc5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34951 } }
I20260620 22:56:21.004069 2849 raft_consensus.cc:388] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:21.004108 2849 raft_consensus.cc:482] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:21.004160 2849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:21.004930 2849 raft_consensus.cc:504] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20b3efb3f134cf99ff144202088cbc5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34951 } }
I20260620 22:56:21.005048 2849 leader_election.cc:302] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d20b3efb3f134cf99ff144202088cbc5; no voters:
I20260620 22:56:21.005232 2849 leader_election.cc:288] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:21.005275 2854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:21.005396 2854 raft_consensus.cc:686] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [term 1 LEADER]: Becoming Leader. State: Replica: d20b3efb3f134cf99ff144202088cbc5, State: Running, Role: LEADER
I20260620 22:56:21.005475 2854 consensus_queue.cc:237] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20b3efb3f134cf99ff144202088cbc5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34951 } }
I20260620 22:56:21.005533 2849 sys_catalog.cc:564] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:21.006726 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d20b3efb3f134cf99ff144202088cbc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20b3efb3f134cf99ff144202088cbc5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34951 } } }
I20260620 22:56:21.006819 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:21.006960 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d20b3efb3f134cf99ff144202088cbc5. Latest consensus state: current_term: 1 leader_uuid: "d20b3efb3f134cf99ff144202088cbc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20b3efb3f134cf99ff144202088cbc5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34951 } } }
I20260620 22:56:21.007015 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5 [sys.catalog]: This master's current role is: LEADER
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:21.007550 2858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:21.008090 2858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:21.010095 2858 catalog_manager.cc:1347] Generated new cluster ID: 8aecebfc98cc4c5388c6e812d5aec14a
I20260620 22:56:21.010138 2858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:21.011829 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:20.999521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.1.166.62:34951)
0620 22:56:20.999827 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:20.999841 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:20.999947 (+ 106us) server_negotiation.cc:407] Connection header received
0620 22:56:21.000712 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.000722 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.000803 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.000973 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:21.001907 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.003042 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.004122 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.004805 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.004905 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.007953 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:21.007957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:21.007958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:21.007974 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:21.009981 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.010460 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.010464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.010464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.010592 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.010832 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.010833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.010833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.011162 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:21.011320 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.011411 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.011573 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":118,"threads_started":1}
I20260620 22:56:21.020011 2858 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:21.020866 2858 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:21.032330 2858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5: Generated new TSK 0
I20260620 22:56:21.032590 2858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:21.061275 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34951
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:21.151942 2875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:21.152261 2875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:21.152295 2875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:21.154605 2875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:21.154711 2875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:21.154745 2875 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:56:21.154769 2875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:21.157096 2875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:34951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:21.157398 2875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:21.157658 2875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:21.162945 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:21.162935 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:21.162935 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:21.163168 2875 server_base.cc:1034] running on GCE node
I20260620 22:56:21.163424 2875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:21.163669 2875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:21.164870 2875 hybrid_clock.cc:648] HybridClock initialized: now 1781996181164852 us; error 39 us; skew 500 ppm
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:21.167804 2875 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:21.168279 2875 webserver.cc:460] Webserver started at http://127.1.166.1:36253/ using document root <none> and password file <none>
I20260620 22:56:21.168504 2875 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:21.168555 2875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:21.168663 2875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:21.169749 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "1c8894d5ad364ad4b67ec7a58d777b21"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "5054b0393f8ff2dd229c619e80605e1a"
server_key_iv: "8266dbe8121fce6d86a3310d715e3b60"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.170073 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "1c8894d5ad364ad4b67ec7a58d777b21"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "5054b0393f8ff2dd229c619e80605e1a"
server_key_iv: "8266dbe8121fce6d86a3310d715e3b60"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.171494 2875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:56:21.172320 2890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.172633 2875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:21.172720 2875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "1c8894d5ad364ad4b67ec7a58d777b21"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "5054b0393f8ff2dd229c619e80605e1a"
server_key_iv: "8266dbe8121fce6d86a3310d715e3b60"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.172793 2875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:21.193919 2875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:21.196200 2875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:21.196326 2875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:21.196887 2875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:21.196926 2875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.196947 2875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:21.196959 2875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.203358 2875 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:34153
I20260620 22:56:21.203506 3005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:34153 every 8 connection(s)
I20260620 22:56:21.203758 2875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:21.206990 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 2875
I20260620 22:56:21.207086 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:21.207264 1688 external_mini_cluster.cc:1428] Setting key 7a7e9a1315a5d8f708b64bb4aa4a7430
I20260620 22:56:21.207764 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.197273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34951 (local address 127.1.166.1:42045)
0620 22:56:21.197458 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.197499 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.197972 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.198223 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.198228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.198589 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:21.198892 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.198899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.199881 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.199882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.201192 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.201196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.201797 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.201798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.201927 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.202652 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:21.202674 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:21.204219 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:21.205819 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.205827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.205829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.206070 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.206278 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.206279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.206279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.206600 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.207019 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:21.207027 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:21.207230 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.207357 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.207535 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260620 22:56:21.208221 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.197298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:42045 (local address 127.1.166.62:34951)
0620 22:56:21.197366 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.197369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.197586 (+ 217us) server_negotiation.cc:407] Connection header received
0620 22:56:21.198067 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.198068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.198080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.198159 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:21.198988 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.199793 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.201347 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.201672 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.201761 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.204309 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:21.204311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:21.204311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:21.204326 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:21.205721 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.206158 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.206159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.206159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.206220 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.206687 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.206688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.206688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.206917 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:21.206990 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.208060 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.208142 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:56:21.209941 3006 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34951
I20260620 22:56:21.210063 3006 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:21.210292 3006 heartbeater.cc:511] Master 127.1.166.62:34951 requested a full tablet report, sending...
I20260620 22:56:21.210958 2802 ts_manager.cc:194] Registered new tserver with Master: 1c8894d5ad364ad4b67ec7a58d777b21 (127.1.166.1:34153)
I20260620 22:56:21.211781 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:42045
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
I20260620 22:56:21.218057 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.211132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34951 (local address 127.1.166.1:46431)
0620 22:56:21.211294 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.211305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.211375 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.211646 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.211647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.212100 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:21.212187 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.212192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.213067 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.213069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.213910 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.213914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.214304 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.214305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.214426 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.214939 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:21.214945 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:21.215413 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:21.216967 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.216968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.216969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.217151 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.217380 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.217382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.217382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.217451 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.217689 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:21.217690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:21.217748 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.217812 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.217937 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260620 22:56:21.218057 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.211196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:46431 (local address 127.1.166.62:34951)
0620 22:56:21.211238 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.211240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.211374 (+ 134us) server_negotiation.cc:407] Connection header received
0620 22:56:21.211539 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.211540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.211552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.211619 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:21.212285 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.212957 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.214012 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.214217 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.214285 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.215537 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:21.215539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:21.215539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:21.215546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:21.216880 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.217219 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.217220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.217220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.217311 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.217510 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.217511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.217511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.217632 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:21.217699 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.217846 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.217936 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:21.232690 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34951
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:21.321884 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:21.322131 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:21.322157 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:21.323592 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:21.323657 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:21.323680 3015 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:56:21.323697 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:21.325392 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:34951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:21.325683 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:21.325932 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:21.331265 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:21.331382 3015 server_base.cc:1034] running on GCE node
W20260620 22:56:21.331296 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:21.331266 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:21.331753 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:21.332000 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:21.333169 3015 hybrid_clock.cc:648] HybridClock initialized: now 1781996181333153 us; error 29 us; skew 500 ppm
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:21.335853 3015 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:21.336297 3015 webserver.cc:460] Webserver started at http://127.1.166.2:46349/ using document root <none> and password file <none>
I20260620 22:56:21.336500 3015 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:21.336549 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:21.336694 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:21.337730 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "0f16da3d6a054d73bc145b75ca8328ac"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "5afad2864f485a2540245821a8623f4f"
server_key_iv: "3b2516de5cefda15d9881a47c65a966f"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.338054 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "0f16da3d6a054d73bc145b75ca8328ac"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "5afad2864f485a2540245821a8623f4f"
server_key_iv: "3b2516de5cefda15d9881a47c65a966f"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.339324 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:21.339977 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.340118 3015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.340198 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "0f16da3d6a054d73bc145b75ca8328ac"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "5afad2864f485a2540245821a8623f4f"
server_key_iv: "3b2516de5cefda15d9881a47c65a966f"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.340258 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:21.358484 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:21.360790 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:21.360941 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:21.361469 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:21.361510 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.361550 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:21.361575 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.368094 3015 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:41799
I20260620 22:56:21.368276 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:41799 every 8 connection(s)
I20260620 22:56:21.368573 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:21.373808 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.362023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:58347 (local address 127.1.166.62:34951)
0620 22:56:21.362070 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.362072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.362332 (+ 260us) server_negotiation.cc:407] Connection header received
0620 22:56:21.362928 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.362930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.362944 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.363018 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:21.363865 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.364646 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.366436 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.366741 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.366825 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.370731 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:21.370733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:21.370733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:21.370742 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:21.372037 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.372593 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.372594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.372595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.372663 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.372957 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.372958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.372959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.373092 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:21.373154 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.373591 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.373715 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:56:21.373832 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.362029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34951 (local address 127.1.166.2:58347)
0620 22:56:21.362200 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.362238 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.362769 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.363105 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.363118 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.363471 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:21.363763 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.363769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.364884 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.364885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.366311 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.366315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.366890 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.366892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.367051 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.368293 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:21.368312 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:21.370595 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:21.372148 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.372160 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.372163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.372468 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.372726 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.372726 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.372727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.372837 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.373175 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:21.373183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:21.373343 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.373452 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.373614 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260620 22:56:21.374390 3146 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34951
I20260620 22:56:21.374501 3146 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:21.374706 3146 heartbeater.cc:511] Master 127.1.166.62:34951 requested a full tablet report, sending...
I20260620 22:56:21.375252 2803 ts_manager.cc:194] Registered new tserver with Master: 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799)
I20260620 22:56:21.375841 2803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:58347
I20260620 22:56:21.378248 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 3015
I20260620 22:56:21.378432 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:21.378624 1688 external_mini_cluster.cc:1428] Setting key 70d0f8ac6562700f6a0e720b82481565
I20260620 22:56:21.382527 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.375969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:52291 (local address 127.1.166.62:34951)
0620 22:56:21.376058 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.376060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.376167 (+ 107us) server_negotiation.cc:407] Connection header received
0620 22:56:21.376231 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.376231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.376245 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.376309 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:21.376822 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.377375 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.378245 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.378647 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.378726 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.380010 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:21.380012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:21.380012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:21.380020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:21.381301 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.381631 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.381632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.381632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.381685 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.381854 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.381855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.381861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.382025 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:21.382086 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.382238 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.382426 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:56:21.382541 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.375912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34951 (local address 127.1.166.2:52291)
0620 22:56:21.376087 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.376098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.376165 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.376329 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.376329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.376575 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:21.376649 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.376652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.377508 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.377509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.378122 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.378125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.378795 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.378797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.378993 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.379564 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:21.379570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:21.379904 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:21.381390 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.381391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.381391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.381565 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.381740 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.381741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.381741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.381803 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.382089 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:21.382090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:21.382146 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.382205 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.382332 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:21.405778 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34951
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:21.492882 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:21.493140 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:21.493167 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:21.494655 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:21.494735 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:21.494760 3155 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:56:21.494781 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:21.496615 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:34951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:21.496932 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:21.497184 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:21.503293 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:21.503304 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:21.503289 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:21.503731 3155 server_base.cc:1034] running on GCE node
I20260620 22:56:21.504031 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:21.504284 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:21.505532 3155 hybrid_clock.cc:648] HybridClock initialized: now 1781996181505503 us; error 39 us; skew 500 ppm
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:21.509482 3155 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:21.510058 3155 webserver.cc:460] Webserver started at http://127.1.166.3:45167/ using document root <none> and password file <none>
I20260620 22:56:21.510325 3155 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:21.510427 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:21.510568 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:21.511780 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "42248a3aeb5540d4a3745ea682782f60"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "270c84924c88868a548c918db0dcd4ae"
server_key_iv: "c39be71f4e6ad6964ef2d958b990994f"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.512166 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "42248a3aeb5540d4a3745ea682782f60"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "270c84924c88868a548c918db0dcd4ae"
server_key_iv: "c39be71f4e6ad6964ef2d958b990994f"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.513680 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:21.514709 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.515059 3155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:21.515183 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "42248a3aeb5540d4a3745ea682782f60"
format_stamp: "Formatted at 2026-06-20 22:56:21 on dist-test-slave-rdl9"
server_key: "270c84924c88868a548c918db0dcd4ae"
server_key_iv: "c39be71f4e6ad6964ef2d958b990994f"
server_key_version: "encryptionkey@0"
I20260620 22:56:21.515268 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:21.545802 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:21.548326 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:21.548496 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:21.549031 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:21.549070 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.549101 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:21.549117 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:21.556424 3155 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:43089
I20260620 22:56:21.556510 3285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:43089 every 8 connection(s)
I20260620 22:56:21.556870 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:21.559298 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.549592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:40583 (local address 127.1.166.62:34951)
0620 22:56:21.549646 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.549648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.549810 (+ 162us) server_negotiation.cc:407] Connection header received
0620 22:56:21.550388 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.550390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.550405 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.550580 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:21.551266 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.552021 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.553066 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.553374 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.553458 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.556248 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:21.556250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:21.556251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:21.556259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:21.557800 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.558268 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.558269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.558270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.558366 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.558644 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.558645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.558646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.558798 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:21.558863 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.559136 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.559221 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:56:21.559418 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.549535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34951 (local address 127.1.166.3:40583)
0620 22:56:21.549696 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.549734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.550224 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.550703 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.550708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.550978 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:21.551180 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.551185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.552125 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.552126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.552993 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.552996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.553476 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.553477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.553620 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.554224 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:21.554234 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:21.556127 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:21.557908 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.557915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.557917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.558159 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.558455 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.558456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.558456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.558561 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.558860 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:21.558864 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:21.558992 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.559076 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.559218 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260620 22:56:21.559885 3286 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34951
I20260620 22:56:21.559978 3286 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:21.560165 3286 heartbeater.cc:511] Master 127.1.166.62:34951 requested a full tablet report, sending...
I20260620 22:56:21.560563 2802 ts_manager.cc:194] Registered new tserver with Master: 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089)
I20260620 22:56:21.561023 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:40583
I20260620 22:56:21.561555 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 3155
I20260620 22:56:21.561645 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:21.561836 1688 external_mini_cluster.cc:1428] Setting key 0d26aeb866a2aca07ea6bba79af6fe84
I20260620 22:56:21.562690 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:56:21.566923 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.561081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:56553 (local address 127.1.166.62:34951)
0620 22:56:21.561155 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.561157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.561241 (+ 84us) server_negotiation.cc:407] Connection header received
0620 22:56:21.561300 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.561301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.561310 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.561355 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:21.561704 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.562328 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.563172 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.563380 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.563438 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.564483 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:21.564485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:21.564485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:21.564492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:21.565740 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.566136 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.566137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.566137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.566201 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.566434 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.566435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.566435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.566568 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:21.566626 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.566787 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.566855 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 22:56:21.567003 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.560990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34951 (local address 127.1.166.3:56553)
0620 22:56:21.561164 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.561176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.561242 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.561358 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.561359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.561549 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:21.561619 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.561621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.562473 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.562474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.563095 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.563098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.563445 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.563446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.563528 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.563992 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:21.563996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:21.564341 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:21.565824 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.565825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.565826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.566039 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.566273 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:21.566273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:21.566274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:21.566371 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:21.566628 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:21.566629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:21.566690 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.566762 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.566885 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:56:21 dist-test-slave-rdl9 krb5kdc[2758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:21.601369 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.589017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.1.166.62:34951)
0620 22:56:21.589087 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.589090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.589106 (+ 16us) server_negotiation.cc:407] Connection header received
0620 22:56:21.589140 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.589141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.589154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.589252 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:21.590059 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.590931 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.592322 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.592666 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.592748 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.595836 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:21.595838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:21.595839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:21.595849 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:21.599949 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.600451 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.600453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.600453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.600535 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:21.600748 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:21.600749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:21.600750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:21.600913 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:21.601002 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.601173 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.601269 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:56:21.603870 2802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:21.605751 2802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:21.613447 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.609058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.1.166.1:34153)
0620 22:56:21.609214 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.609218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.609227 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:21.609419 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.609421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.609436 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.609671 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:21.609730 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.610936 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.612327 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.612923 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.613011 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.613082 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.613266 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.613314 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260620 22:56:21.613453 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.608836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:34153 (local address 127.0.0.1:55252)
0620 22:56:21.609013 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.609023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.609079 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.609556 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.609557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.609558 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:21.609596 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.609598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.611045 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.611046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.612223 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.612226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.613035 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.613036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.613150 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.613168 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.613244 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.613314 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260620 22:56:21.613651 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.608726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.1.166.3:43089)
0620 22:56:21.608946 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.608952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.608965 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:56:21.609123 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.609126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.609142 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.609191 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:21.609442 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.610538 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.612216 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.613008 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.613088 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.613148 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.613503 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.613546 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
I20260620 22:56:21.613745 3304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.608630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:43089 (local address 127.0.0.1:45710)
0620 22:56:21.608829 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.608878 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.608959 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.609276 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.609279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.609281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:21.609351 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.609353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.610640 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.610642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.612086 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.612091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.613188 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.613189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.613266 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.613270 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.613314 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.613369 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":62,"threads_started":1}
I20260620 22:56:21.613904 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.608635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:41799 (local address 127.0.0.1:49128)
0620 22:56:21.608939 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.608948 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.608996 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.609492 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.609493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.609496 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:21.609597 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.609598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.611043 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.611044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.612354 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.612357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.613347 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.613348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.613403 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.613407 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.613452 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.613492 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":36,"thread_start_us":107,"threads_started":1}
I20260620 22:56:21.614073 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.608782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.1.166.2:41799)
0620 22:56:21.609257 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.609261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.609271 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:21.609312 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.609314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.609341 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.609416 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:21.609675 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.610936 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.612460 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.613193 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.613251 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.613299 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.613857 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.613906 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":78,"threads_started":1}
I20260620 22:56:21.614532 3220 tablet_service.cc:1467] Processing CreateTablet for tablet 06af64a266ca49d79531c223a4e39a70 (DEFAULT_TABLE table=test-table [id=169beb7db3e04734ada182b61eb1d8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:21.614540 2937 tablet_service.cc:1467] Processing CreateTablet for tablet 06af64a266ca49d79531c223a4e39a70 (DEFAULT_TABLE table=test-table [id=169beb7db3e04734ada182b61eb1d8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:21.614909 3220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06af64a266ca49d79531c223a4e39a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:21.614933 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06af64a266ca49d79531c223a4e39a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:21.614990 3080 tablet_service.cc:1467] Processing CreateTablet for tablet 06af64a266ca49d79531c223a4e39a70 (DEFAULT_TABLE table=test-table [id=169beb7db3e04734ada182b61eb1d8d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:21.615339 3080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06af64a266ca49d79531c223a4e39a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:21.617486 3309 tablet_bootstrap.cc:492] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Bootstrap starting.
I20260620 22:56:21.618472 3309 tablet_bootstrap.cc:654] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:21.618762 3309 log.cc:824] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:21.618916 3310 tablet_bootstrap.cc:492] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Bootstrap starting.
I20260620 22:56:21.619370 3311 tablet_bootstrap.cc:492] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: Bootstrap starting.
I20260620 22:56:21.619505 3310 tablet_bootstrap.cc:654] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:21.619786 3310 log.cc:824] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:21.619931 3311 tablet_bootstrap.cc:654] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:21.620195 3311 log.cc:824] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:21.620499 3309 tablet_bootstrap.cc:492] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: No bootstrap required, opened a new log
I20260620 22:56:21.620565 3309 ts_tablet_manager.cc:1397] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 22:56:21.620859 3311 tablet_bootstrap.cc:492] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: No bootstrap required, opened a new log
I20260620 22:56:21.620918 3311 ts_tablet_manager.cc:1397] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:56:21.622000 3309 raft_consensus.cc:348] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.622130 3309 raft_consensus.cc:374] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:21.622159 3309 raft_consensus.cc:729] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42248a3aeb5540d4a3745ea682782f60, State: Initialized, Role: FOLLOWER
I20260620 22:56:21.622272 3309 consensus_queue.cc:260] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.622399 3311 raft_consensus.cc:348] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.622565 3311 raft_consensus.cc:374] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:21.622581 3309 ts_tablet_manager.cc:1428] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 22:56:21.622618 3311 raft_consensus.cc:729] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f16da3d6a054d73bc145b75ca8328ac, State: Initialized, Role: FOLLOWER
I20260620 22:56:21.622715 3310 tablet_bootstrap.cc:492] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: No bootstrap required, opened a new log
I20260620 22:56:21.622779 3310 ts_tablet_manager.cc:1397] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 22:56:21.622751 3311 consensus_queue.cc:260] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.622822 3286 heartbeater.cc:503] Master 127.1.166.62:34951 was elected leader, sending a full tablet report...
I20260620 22:56:21.623016 3146 heartbeater.cc:503] Master 127.1.166.62:34951 was elected leader, sending a full tablet report...
I20260620 22:56:21.623247 3311 ts_tablet_manager.cc:1428] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:56:21.624541 3310 raft_consensus.cc:348] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.624663 3310 raft_consensus.cc:374] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:21.624682 3310 raft_consensus.cc:729] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8894d5ad364ad4b67ec7a58d777b21, State: Initialized, Role: FOLLOWER
I20260620 22:56:21.624758 3310 consensus_queue.cc:260] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.624982 3310 ts_tablet_manager.cc:1428] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:56:21.625041 3006 heartbeater.cc:503] Master 127.1.166.62:34951 was elected leader, sending a full tablet report...
W20260620 22:56:21.704663 3007 tablet.cc:2369] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:21.782584 3317 raft_consensus.cc:482] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:21.782716 3317 raft_consensus.cc:504] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.783102 3317 leader_election.cc:288] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799), 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089)
I20260620 22:56:21.787092 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.783328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:43089 (local address 127.1.166.1:33133)
0620 22:56:21.783529 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.783538 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.783598 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.783731 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.783732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.783734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:21.783908 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.783911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.784817 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.784818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.786094 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.786100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.786781 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.786782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.786871 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.786876 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.786950 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.787005 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":30,"thread_start_us":59,"threads_started":1}
I20260620 22:56:21.787112 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.783403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:33133 (local address 127.1.166.3:43089)
0620 22:56:21.783459 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.783461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.783605 (+ 144us) server_negotiation.cc:407] Connection header received
0620 22:56:21.783656 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.783657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.783668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.783722 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:21.784025 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.784622 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.786216 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.786693 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.786752 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.786772 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.786978 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.787033 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:56:21.787443 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.783419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:56151 (local address 127.1.166.2:41799)
0620 22:56:21.783468 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.783469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.783528 (+ 59us) server_negotiation.cc:407] Connection header received
0620 22:56:21.783595 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.783596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.783613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.783664 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:21.783911 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.785257 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.786466 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.786930 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.786999 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.787008 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.787321 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.787374 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:56:21.787535 3240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06af64a266ca49d79531c223a4e39a70" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42248a3aeb5540d4a3745ea682782f60" is_pre_election: true
I20260620 22:56:21.787673 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.783288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:41799 (local address 127.1.166.1:56151)
0620 22:56:21.783445 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:21.783457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:21.783526 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:21.783681 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:21.783682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:21.783684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:21.783752 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.783754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.785443 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.785444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.786317 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:21.786319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.787029 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.787030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:21.787119 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.787124 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:56:21.787538 (+ 414us) client_negotiation.cc:240] Negotiation successful
0620 22:56:21.787593 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":33,"thread_start_us":71,"threads_started":1}
I20260620 22:56:21.787705 3240 raft_consensus.cc:2454] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 0.
I20260620 22:56:21.787990 2894 leader_election.cc:302] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8894d5ad364ad4b67ec7a58d777b21, 42248a3aeb5540d4a3745ea682782f60; no voters:
I20260620 22:56:21.788018 3100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06af64a266ca49d79531c223a4e39a70" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f16da3d6a054d73bc145b75ca8328ac" is_pre_election: true
I20260620 22:56:21.788174 3100 raft_consensus.cc:2454] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 0.
I20260620 22:56:21.788219 3317 raft_consensus.cc:2781] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:21.788300 3317 raft_consensus.cc:482] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:21.788336 3317 raft_consensus.cc:3037] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:21.789101 3317 raft_consensus.cc:504] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.789335 3317 leader_election.cc:288] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 election: Requested vote from peers 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799), 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089)
I20260620 22:56:21.789435 3240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06af64a266ca49d79531c223a4e39a70" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42248a3aeb5540d4a3745ea682782f60"
I20260620 22:56:21.789443 3100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06af64a266ca49d79531c223a4e39a70" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f16da3d6a054d73bc145b75ca8328ac"
I20260620 22:56:21.789523 3240 raft_consensus.cc:3037] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:21.789539 3100 raft_consensus.cc:3037] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:21.790237 3100 raft_consensus.cc:2454] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 1.
I20260620 22:56:21.790251 3240 raft_consensus.cc:2454] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 1.
I20260620 22:56:21.790477 2891 leader_election.cc:302] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f16da3d6a054d73bc145b75ca8328ac, 1c8894d5ad364ad4b67ec7a58d777b21; no voters:
I20260620 22:56:21.790593 3317 raft_consensus.cc:2781] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:21.790719 3317 raft_consensus.cc:686] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 LEADER]: Becoming Leader. State: Replica: 1c8894d5ad364ad4b67ec7a58d777b21, State: Running, Role: LEADER
I20260620 22:56:21.790817 3317 consensus_queue.cc:237] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } }
I20260620 22:56:21.791697 2802 catalog_manager.cc:5495] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c8894d5ad364ad4b67ec7a58d777b21 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } health_report { overall_health: HEALTHY } } }
I20260620 22:56:21.802795 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:21.800204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.1.166.1:34153)
0620 22:56:21.800303 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:21.800305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:21.800316 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:21.800371 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:21.800372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:21.800392 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:21.800456 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:21.800688 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.801158 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.802000 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:21.802187 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:21.802246 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:21.802451 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:21.802549 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:21.802608 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:21.802700 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 22:56:21.802720 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 22:56:21.804736 3240 raft_consensus.cc:1261] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Refusing update from remote peer 1c8894d5ad364ad4b67ec7a58d777b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:21.804893 3100 raft_consensus.cc:1261] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Refusing update from remote peer 1c8894d5ad364ad4b67ec7a58d777b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:21.805198 3320 consensus_queue.cc:1035] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:21.805361 3317 consensus_queue.cc:1035] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:56:21.807372 3287 tablet.cc:2369] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:21.808436 3325 mvcc.cc:205] Tried to move back new op lower bound from 7299056360669290496 to 7299056360615776256. Current Snapshot: MvccSnapshot[applied={T|T < 7299056360669290496}]
I20260620 22:56:21.808960 3323 mvcc.cc:205] Tried to move back new op lower bound from 7299056360669290496 to 7299056360615776256. Current Snapshot: MvccSnapshot[applied={T|T < 7299056360669290496 or (T in {7299056360669290496})}]
I20260620 22:56:21.812254 2801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:21.812367 2801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:21.813326 2801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5: Sending DeleteTablet for 3 replicas of tablet 06af64a266ca49d79531c223a4e39a70
I20260620 22:56:21.813716 3080 tablet_service.cc:1514] Processing DeleteTablet for tablet 06af64a266ca49d79531c223a4e39a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:21 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:49128
I20260620 22:56:21.813766 2937 tablet_service.cc:1514] Processing DeleteTablet for tablet 06af64a266ca49d79531c223a4e39a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:21 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:55252
I20260620 22:56:21.813745 3220 tablet_service.cc:1514] Processing DeleteTablet for tablet 06af64a266ca49d79531c223a4e39a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:21 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:45710
I20260620 22:56:21.814018 3334 tablet_replica.cc:330] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: stopping tablet replica
I20260620 22:56:21.814147 3334 raft_consensus.cc:2229] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:21.814293 3334 raft_consensus.cc:2258] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:21.814393 2801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 22:56:21.814567 3333 tablet_replica.cc:330] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: stopping tablet replica
W20260620 22:56:21.814651 2801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:21.814698 3333 raft_consensus.cc:2229] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:21.814815 3333 raft_consensus.cc:2258] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:21.814252 3332 tablet_replica.cc:330] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: stopping tablet replica
I20260620 22:56:21.815255 3332 raft_consensus.cc:2229] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:56:21.815316 3334 ts_tablet_manager.cc:1905] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:21.816236 3332 raft_consensus.cc:2258] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:21.816570 2937 tablet_service.cc:1467] Processing CreateTablet for tablet 3994a8be53ee430bacde17468889f2bc (DEFAULT_TABLE table=test-table [id=73b907a3dd114e23bdb829ad4d21ef8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:21.816699 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3994a8be53ee430bacde17468889f2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:21.816841 3332 ts_tablet_manager.cc:1905] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:21.817530 3334 ts_tablet_manager.cc:1918] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:21.817612 3334 log.cc:1196] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/06af64a266ca49d79531c223a4e39a70
I20260620 22:56:21.817916 3334 ts_tablet_manager.cc:1939] T 06af64a266ca49d79531c223a4e39a70 P 42248a3aeb5540d4a3745ea682782f60: Deleting consensus metadata
I20260620 22:56:21.818274 3310 tablet_bootstrap.cc:492] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: Bootstrap starting.
I20260620 22:56:21.818760 3310 tablet_bootstrap.cc:654] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:21.818723 3333 ts_tablet_manager.cc:1905] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:21.818910 3332 ts_tablet_manager.cc:1918] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:21.818974 3332 log.cc:1196] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/06af64a266ca49d79531c223a4e39a70
I20260620 22:56:21.819208 3332 ts_tablet_manager.cc:1939] T 06af64a266ca49d79531c223a4e39a70 P 1c8894d5ad364ad4b67ec7a58d777b21: Deleting consensus metadata
I20260620 22:56:21.819180 2790 catalog_manager.cc:4841] TS 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089): tablet 06af64a266ca49d79531c223a4e39a70 (table test-table [id=169beb7db3e04734ada182b61eb1d8d2]) successfully deleted
I20260620 22:56:21.819528 3220 tablet_service.cc:1467] Processing CreateTablet for tablet 3994a8be53ee430bacde17468889f2bc (DEFAULT_TABLE table=test-table [id=73b907a3dd114e23bdb829ad4d21ef8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:21.819640 3220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3994a8be53ee430bacde17468889f2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:21.819989 2790 catalog_manager.cc:4841] TS 1c8894d5ad364ad4b67ec7a58d777b21 (127.1.166.1:34153): tablet 06af64a266ca49d79531c223a4e39a70 (table test-table [id=169beb7db3e04734ada182b61eb1d8d2]) successfully deleted
I20260620 22:56:21.820569 3333 ts_tablet_manager.cc:1918] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:21.820622 3333 log.cc:1196] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/06af64a266ca49d79531c223a4e39a70
I20260620 22:56:21.820782 3333 ts_tablet_manager.cc:1939] T 06af64a266ca49d79531c223a4e39a70 P 0f16da3d6a054d73bc145b75ca8328ac: Deleting consensus metadata
I20260620 22:56:21.821012 3309 tablet_bootstrap.cc:492] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: Bootstrap starting.
I20260620 22:56:21.821225 2787 catalog_manager.cc:4841] TS 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799): tablet 06af64a266ca49d79531c223a4e39a70 (table test-table [id=169beb7db3e04734ada182b61eb1d8d2]) successfully deleted
I20260620 22:56:21.821280 3080 tablet_service.cc:1467] Processing CreateTablet for tablet 3994a8be53ee430bacde17468889f2bc (DEFAULT_TABLE table=test-table [id=73b907a3dd114e23bdb829ad4d21ef8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:21.821382 3080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3994a8be53ee430bacde17468889f2bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:21.821406 3309 tablet_bootstrap.cc:654] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:21.821942 3309 tablet_bootstrap.cc:492] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: No bootstrap required, opened a new log
I20260620 22:56:21.821974 3309 ts_tablet_manager.cc:1397] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:56:21.822091 3309 raft_consensus.cc:348] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:21.822144 3309 raft_consensus.cc:374] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:21.822155 3309 raft_consensus.cc:729] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42248a3aeb5540d4a3745ea682782f60, State: Initialized, Role: FOLLOWER
I20260620 22:56:21.822196 3309 consensus_queue.cc:260] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:21.822284 3309 ts_tablet_manager.cc:1428] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:21.822585 3311 tablet_bootstrap.cc:492] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: Bootstrap starting.
I20260620 22:56:21.823007 3311 tablet_bootstrap.cc:654] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:21.823613 3311 tablet_bootstrap.cc:492] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: No bootstrap required, opened a new log
I20260620 22:56:21.823648 3310 tablet_bootstrap.cc:492] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: No bootstrap required, opened a new log
I20260620 22:56:21.823678 3311 ts_tablet_manager.cc:1397] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:21.823693 3310 ts_tablet_manager.cc:1397] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 22:56:21.823830 3310 raft_consensus.cc:348] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:21.823829 3311 raft_consensus.cc:348] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:21.823905 3310 raft_consensus.cc:374] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:21.823905 3311 raft_consensus.cc:374] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:21.823926 3310 raft_consensus.cc:729] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8894d5ad364ad4b67ec7a58d777b21, State: Initialized, Role: FOLLOWER
I20260620 22:56:21.823926 3311 raft_consensus.cc:729] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f16da3d6a054d73bc145b75ca8328ac, State: Initialized, Role: FOLLOWER
I20260620 22:56:21.823971 3311 consensus_queue.cc:260] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:21.823969 3310 consensus_queue.cc:260] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:21.824056 3311 ts_tablet_manager.cc:1428] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:21.824056 3310 ts_tablet_manager.cc:1428] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260620 22:56:21.869357 3147 tablet.cc:2369] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:22.012802 3317 raft_consensus.cc:482] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:22.012905 3317 raft_consensus.cc:504] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.013077 3317 leader_election.cc:288] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799), 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089)
I20260620 22:56:22.013324 3240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3994a8be53ee430bacde17468889f2bc" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42248a3aeb5540d4a3745ea682782f60" is_pre_election: true
I20260620 22:56:22.013324 3100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3994a8be53ee430bacde17468889f2bc" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f16da3d6a054d73bc145b75ca8328ac" is_pre_election: true
I20260620 22:56:22.013430 3100 raft_consensus.cc:2454] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 0.
I20260620 22:56:22.013430 3240 raft_consensus.cc:2454] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 0.
I20260620 22:56:22.013574 2894 leader_election.cc:302] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8894d5ad364ad4b67ec7a58d777b21, 42248a3aeb5540d4a3745ea682782f60; no voters:
I20260620 22:56:22.013681 3317 raft_consensus.cc:2781] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:22.013721 3317 raft_consensus.cc:482] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:22.013738 3317 raft_consensus.cc:3037] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:22.014406 3317 raft_consensus.cc:504] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.014530 3317 leader_election.cc:288] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 election: Requested vote from peers 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799), 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089)
I20260620 22:56:22.014716 3240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3994a8be53ee430bacde17468889f2bc" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42248a3aeb5540d4a3745ea682782f60"
I20260620 22:56:22.014716 3100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3994a8be53ee430bacde17468889f2bc" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f16da3d6a054d73bc145b75ca8328ac"
I20260620 22:56:22.014792 3240 raft_consensus.cc:3037] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:22.014792 3100 raft_consensus.cc:3037] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:22.015523 3240 raft_consensus.cc:2454] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 1.
I20260620 22:56:22.015523 3100 raft_consensus.cc:2454] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 1.
I20260620 22:56:22.015679 2894 leader_election.cc:302] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8894d5ad364ad4b67ec7a58d777b21, 42248a3aeb5540d4a3745ea682782f60; no voters:
I20260620 22:56:22.015812 3317 raft_consensus.cc:2781] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:22.015867 3317 raft_consensus.cc:686] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 LEADER]: Becoming Leader. State: Replica: 1c8894d5ad364ad4b67ec7a58d777b21, State: Running, Role: LEADER
I20260620 22:56:22.015918 3317 consensus_queue.cc:237] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.016467 2801 catalog_manager.cc:5495] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c8894d5ad364ad4b67ec7a58d777b21 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } health_report { overall_health: UNKNOWN } } }
Jun 20 22:56:22 dist-test-slave-rdl9 krb5kdc[2758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:22.075224 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:22.067635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.1.166.62:34951)
0620 22:56:22.067703 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:22.067705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:22.067857 (+ 152us) server_negotiation.cc:407] Connection header received
0620 22:56:22.067893 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:22.067894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:22.067906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:22.067987 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:22.068545 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.069109 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.069857 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.070040 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.070099 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:22.072703 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:22.072705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:22.072706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:22.072714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:22.074168 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:22.074556 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:22.074557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:22.074558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:22.074638 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:22.074801 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:22.074802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:22.074802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:22.074910 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:22.074954 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:22.075068 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 22:56:22.075145 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:56:22.075318 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:22.067448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34951 (local address 127.0.0.1:37452)
0620 22:56:22.067711 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:22.067725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:22.067832 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:22.068044 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:22.068045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:22.068367 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:22.068424 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:22.068427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.069209 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.069211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:22.069755 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:22.069757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.070154 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.070155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:22.070210 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:22.070894 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:22.070900 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:22.072583 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:22.074247 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:22.074255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:22.074257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:22.074500 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:22.074710 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:22.074710 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:22.074711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:22.074764 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:22.074952 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:22.074958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:22.075002 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 22:56:22.075050 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:56:22.075204 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":43,"thread_start_us":91,"threads_started":1}
I20260620 22:56:22.077008 2801 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:37452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:56:22.077183 2801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:22.078415 2937 tablet_service.cc:1467] Processing CreateTablet for tablet a8f24a226fcf431bb3bfef082ad682c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a695e2479a594168a9eaf08d22baeaa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:56:22.078447 3080 tablet_service.cc:1467] Processing CreateTablet for tablet a8f24a226fcf431bb3bfef082ad682c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a695e2479a594168a9eaf08d22baeaa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:56:22.078471 3220 tablet_service.cc:1467] Processing CreateTablet for tablet a8f24a226fcf431bb3bfef082ad682c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a695e2479a594168a9eaf08d22baeaa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:56:22.078569 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f24a226fcf431bb3bfef082ad682c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:22.078569 3080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f24a226fcf431bb3bfef082ad682c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:22.078619 3220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f24a226fcf431bb3bfef082ad682c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:22.079919 3309 tablet_bootstrap.cc:492] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: Bootstrap starting.
I20260620 22:56:22.080008 3310 tablet_bootstrap.cc:492] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: Bootstrap starting.
I20260620 22:56:22.080008 3311 tablet_bootstrap.cc:492] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: Bootstrap starting.
I20260620 22:56:22.080318 3310 tablet_bootstrap.cc:654] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:22.080312 3309 tablet_bootstrap.cc:654] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:22.080376 3311 tablet_bootstrap.cc:654] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:22.080920 3309 tablet_bootstrap.cc:492] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: No bootstrap required, opened a new log
I20260620 22:56:22.080924 3311 tablet_bootstrap.cc:492] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: No bootstrap required, opened a new log
I20260620 22:56:22.080965 3311 ts_tablet_manager.cc:1397] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:22.080967 3309 ts_tablet_manager.cc:1397] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:56:22.081085 3310 tablet_bootstrap.cc:492] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: No bootstrap required, opened a new log
I20260620 22:56:22.081084 3311 raft_consensus.cc:348] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.081121 3310 ts_tablet_manager.cc:1397] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:22.081132 3311 raft_consensus.cc:374] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:22.081144 3311 raft_consensus.cc:729] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f16da3d6a054d73bc145b75ca8328ac, State: Initialized, Role: FOLLOWER
I20260620 22:56:22.081102 3309 raft_consensus.cc:348] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.081178 3309 raft_consensus.cc:374] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:22.081177 3311 consensus_queue.cc:260] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.081199 3309 raft_consensus.cc:729] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42248a3aeb5540d4a3745ea682782f60, State: Initialized, Role: FOLLOWER
I20260620 22:56:22.081210 3310 raft_consensus.cc:348] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.081245 3309 consensus_queue.cc:260] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.081265 3310 raft_consensus.cc:374] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:22.081264 3311 ts_tablet_manager.cc:1428] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:22.081286 3310 raft_consensus.cc:729] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8894d5ad364ad4b67ec7a58d777b21, State: Initialized, Role: FOLLOWER
I20260620 22:56:22.081331 3309 ts_tablet_manager.cc:1428] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:22.081323 3310 consensus_queue.cc:260] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.081396 3310 ts_tablet_manager.cc:1428] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:22.081391 3315 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.081492 3315 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:22.081573 3316 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.081661 3316 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:22.081780 3317 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.081843 3317 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:22.143081 3317 raft_consensus.cc:482] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:22.143186 3317 raft_consensus.cc:504] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.143389 3317 leader_election.cc:288] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799), 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089)
I20260620 22:56:22.143621 3100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f24a226fcf431bb3bfef082ad682c5" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f16da3d6a054d73bc145b75ca8328ac" is_pre_election: true
I20260620 22:56:22.143621 3240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f24a226fcf431bb3bfef082ad682c5" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42248a3aeb5540d4a3745ea682782f60" is_pre_election: true
I20260620 22:56:22.143733 3100 raft_consensus.cc:2454] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 0.
I20260620 22:56:22.143733 3240 raft_consensus.cc:2454] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 0.
I20260620 22:56:22.143890 2894 leader_election.cc:302] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8894d5ad364ad4b67ec7a58d777b21, 42248a3aeb5540d4a3745ea682782f60; no voters:
I20260620 22:56:22.144012 3317 raft_consensus.cc:2781] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:22.144062 3317 raft_consensus.cc:482] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:22.144080 3317 raft_consensus.cc:3037] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:22.144721 3317 raft_consensus.cc:504] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.144865 3317 leader_election.cc:288] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 election: Requested vote from peers 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799), 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089)
I20260620 22:56:22.145149 3100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f24a226fcf431bb3bfef082ad682c5" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f16da3d6a054d73bc145b75ca8328ac"
I20260620 22:56:22.145238 3100 raft_consensus.cc:3037] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:22.145196 3240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f24a226fcf431bb3bfef082ad682c5" candidate_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42248a3aeb5540d4a3745ea682782f60"
I20260620 22:56:22.145375 3240 raft_consensus.cc:3037] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:22.145902 3100 raft_consensus.cc:2454] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 1.
I20260620 22:56:22.146102 2891 leader_election.cc:302] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f16da3d6a054d73bc145b75ca8328ac, 1c8894d5ad364ad4b67ec7a58d777b21; no voters:
I20260620 22:56:22.146199 3317 raft_consensus.cc:2781] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:22.146250 3317 raft_consensus.cc:686] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 LEADER]: Becoming Leader. State: Replica: 1c8894d5ad364ad4b67ec7a58d777b21, State: Running, Role: LEADER
I20260620 22:56:22.146251 3240 raft_consensus.cc:2454] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8894d5ad364ad4b67ec7a58d777b21 in term 1.
I20260620 22:56:22.146299 3317 consensus_queue.cc:237] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } }
I20260620 22:56:22.146891 3320 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: TxnStatusTablet state changed. Reason: New leader 1c8894d5ad364ad4b67ec7a58d777b21. Latest consensus state: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.147141 3320 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:56:22.147229 2802 catalog_manager.cc:5495] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c8894d5ad364ad4b67ec7a58d777b21 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } health_report { overall_health: UNKNOWN } } }
I20260620 22:56:22.148540 3349 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:56:22.148684 3349 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:56:22.161007 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:22.157982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.1.166.1:34153)
0620 22:56:22.158030 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:22.158033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:22.158084 (+ 51us) server_negotiation.cc:407] Connection header received
0620 22:56:22.158193 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:22.158194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:22.158204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:22.158241 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:22.158542 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.159023 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.160372 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.160568 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.160620 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:22.160795 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:22.160823 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:22.160855 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:22.160924 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 22:56:22.160943 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:56:22.161075 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:22.157936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:34153 (local address 127.0.0.1:55278)
0620 22:56:22.158031 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:22.158043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:22.158101 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:22.158267 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:22.158268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:22.158269 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:56:22.158331 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:22.158333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.159119 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.159121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:22.160263 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:22.160269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.160632 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.160633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:22.160690 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:22.160716 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:56:22.160868 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:56:22.160874 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:56:22.160921 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:56:22.160987 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 22:56:22.162184 3100 raft_consensus.cc:1261] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Refusing update from remote peer 1c8894d5ad364ad4b67ec7a58d777b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:22.162186 3240 raft_consensus.cc:1261] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Refusing update from remote peer 1c8894d5ad364ad4b67ec7a58d777b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:22.162451 3317 consensus_queue.cc:1035] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:22.162578 3320 consensus_queue.cc:1035] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:22.163105 3316 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: TxnStatusTablet state changed. Reason: New leader 1c8894d5ad364ad4b67ec7a58d777b21. Latest consensus state: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.163200 3316 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:22.163305 3315 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: TxnStatusTablet state changed. Reason: New leader 1c8894d5ad364ad4b67ec7a58d777b21. Latest consensus state: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.163408 3315 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:22.163775 3320 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.163866 3320 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:56:22.164433 3316 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.164518 3316 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 0f16da3d6a054d73bc145b75ca8328ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:22.164983 3354 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.165107 3354 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 1c8894d5ad364ad4b67ec7a58d777b21: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:56:22.165162 3324 mvcc.cc:205] Tried to move back new op lower bound from 7299056362134220800 to 7299056362072489984. Current Snapshot: MvccSnapshot[applied={T|T < 7299056362134220800}]
I20260620 22:56:22.165331 3315 tablet_replica.cc:439] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8894d5ad364ad4b67ec7a58d777b21" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 34153 } } peers { permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 } } peers { permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 } } }
I20260620 22:56:22.165485 3315 tablet_replica.cc:442] T a8f24a226fcf431bb3bfef082ad682c5 P 42248a3aeb5540d4a3745ea682782f60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:22.172495 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:22.169366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:34153 (local address 127.1.166.1:55717)
0620 22:56:22.169505 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:22.169516 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:22.169569 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:22.169708 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:22.169709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:22.169710 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:56:22.169766 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:22.169768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.170804 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.170805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:22.171393 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:22.171395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.171950 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.171950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:22.172009 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:22.172031 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:56:22.172260 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:56:22.172263 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:56:22.172329 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 22:56:22.172382 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260620 22:56:22.172503 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:22.169436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:55717 (local address 127.1.166.1:34153)
0620 22:56:22.169481 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:22.169483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:22.169571 (+ 88us) server_negotiation.cc:407] Connection header received
0620 22:56:22.169608 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:22.169609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:22.169618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:22.169706 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:22.169817 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.170663 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.171509 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.171842 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.171919 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:22.172124 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:22.172169 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:22.172227 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:22.172343 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 22:56:22.172370 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:56:22.175899 3100 raft_consensus.cc:1261] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Refusing update from remote peer 1c8894d5ad364ad4b67ec7a58d777b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:22.175904 3240 raft_consensus.cc:1261] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Refusing update from remote peer 1c8894d5ad364ad4b67ec7a58d777b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:22.176159 3317 consensus_queue.cc:1035] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42248a3aeb5540d4a3745ea682782f60" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 43089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:22.176287 3320 consensus_queue.cc:1035] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f16da3d6a054d73bc145b75ca8328ac" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:22.190476 2802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:22.190584 2802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:22.191208 2802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d20b3efb3f134cf99ff144202088cbc5: Sending DeleteTablet for 3 replicas of tablet 3994a8be53ee430bacde17468889f2bc
I20260620 22:56:22.191519 2940 tablet_service.cc:1514] Processing DeleteTablet for tablet 3994a8be53ee430bacde17468889f2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:22 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:55252
I20260620 22:56:22.191527 3080 tablet_service.cc:1514] Processing DeleteTablet for tablet 3994a8be53ee430bacde17468889f2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:22 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:49128
I20260620 22:56:22.191519 3220 tablet_service.cc:1514] Processing DeleteTablet for tablet 3994a8be53ee430bacde17468889f2bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:22 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:45710
I20260620 22:56:22.191617 3333 tablet_replica.cc:330] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: stopping tablet replica
I20260620 22:56:22.191628 3332 tablet_replica.cc:330] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: stopping tablet replica
I20260620 22:56:22.191663 3333 raft_consensus.cc:2229] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:22.191684 3332 raft_consensus.cc:2229] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:56:22.191699 3333 raft_consensus.cc:2258] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:22.191738 3332 raft_consensus.cc:2258] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:22.191812 3334 tablet_replica.cc:330] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: stopping tablet replica
I20260620 22:56:22.191854 3334 raft_consensus.cc:2229] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:22.191887 3334 raft_consensus.cc:2258] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:22.192078 3332 ts_tablet_manager.cc:1905] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:22.192235 3334 ts_tablet_manager.cc:1905] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:22.193421 3332 ts_tablet_manager.cc:1918] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:22.193482 3332 log.cc:1196] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/3994a8be53ee430bacde17468889f2bc
I20260620 22:56:22.193527 3334 ts_tablet_manager.cc:1918] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:22.193569 3334 log.cc:1196] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/3994a8be53ee430bacde17468889f2bc
I20260620 22:56:22.193682 3332 ts_tablet_manager.cc:1939] T 3994a8be53ee430bacde17468889f2bc P 1c8894d5ad364ad4b67ec7a58d777b21: Deleting consensus metadata
I20260620 22:56:22.193739 3334 ts_tablet_manager.cc:1939] T 3994a8be53ee430bacde17468889f2bc P 42248a3aeb5540d4a3745ea682782f60: Deleting consensus metadata
I20260620 22:56:22.194199 3333 ts_tablet_manager.cc:1905] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:22.195683 3333 ts_tablet_manager.cc:1918] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:22.195755 3333 log.cc:1196] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/3994a8be53ee430bacde17468889f2bc
I20260620 22:56:22.195933 3333 ts_tablet_manager.cc:1939] T 3994a8be53ee430bacde17468889f2bc P 0f16da3d6a054d73bc145b75ca8328ac: Deleting consensus metadata
I20260620 22:56:22.196298 2787 catalog_manager.cc:4841] TS 0f16da3d6a054d73bc145b75ca8328ac (127.1.166.2:41799): tablet 3994a8be53ee430bacde17468889f2bc (table test-table [id=73b907a3dd114e23bdb829ad4d21ef8c]) successfully deleted
I20260620 22:56:22.196306 2790 catalog_manager.cc:4841] TS 42248a3aeb5540d4a3745ea682782f60 (127.1.166.3:43089): tablet 3994a8be53ee430bacde17468889f2bc (table test-table [id=73b907a3dd114e23bdb829ad4d21ef8c]) successfully deleted
I20260620 22:56:22.196532 2790 catalog_manager.cc:4841] TS 1c8894d5ad364ad4b67ec7a58d777b21 (127.1.166.1:34153): tablet 3994a8be53ee430bacde17468889f2bc (table test-table [id=73b907a3dd114e23bdb829ad4d21ef8c]) successfully deleted
Jun 20 22:56:22 dist-test-slave-rdl9 krb5kdc[2758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:22.206205 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:22.196525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.1.166.1:34153)
0620 22:56:22.196575 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:22.196578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:22.197181 (+ 603us) server_negotiation.cc:407] Connection header received
0620 22:56:22.197244 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:22.197245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:22.197259 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:22.197309 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:22.197908 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.198468 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.199530 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.199731 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.199801 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:22.203151 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:22.203161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:22.203162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:22.203169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:22.204943 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:22.205352 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:22.205356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:22.205356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:22.205425 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:22.205600 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:22.205601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:22.205602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:22.205783 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:22.205929 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:22.206015 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 22:56:22.206107 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 22:56:22.206748 3000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55280
I20260620 22:56:22.214013 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:22.207722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.1.166.62:34951)
0620 22:56:22.207772 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:22.207774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:22.207939 (+ 165us) server_negotiation.cc:407] Connection header received
0620 22:56:22.208012 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:22.208013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:22.208024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:22.208095 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:22.208604 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.209120 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.209906 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:22.210111 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:22.210175 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:22.211482 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:22.211484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:22.211485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:22.211494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:22.212873 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:22.213234 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:22.213235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:22.213236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:22.213295 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:22.213474 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:22.213475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:22.213475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:22.213599 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:22.213699 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:22.213828 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:56:22.213913 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 22:56:22.214501 2802 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37454
I20260620 22:56:22.215407 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 2875
I20260620 22:56:22.221866 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 3015
I20260620 22:56:22.226969 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 3155
I20260620 22:56:22.231315 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 2771
2026-06-20T22:56:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2462 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:22 dist-test-slave-rdl9 krb5kdc[3367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:22 dist-test-slave-rdl9 krb5kdc[3367](info): set up 2 sockets
Jun 20 22:56:22 dist-test-slave-rdl9 krb5kdc[3367](info): commencing operation
krb5kdc: starting...
W20260620 22:56:23.210084 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:23 dist-test-slave-rdl9 krb5kdc[3367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:23.331171 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38215
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:23.418179 3380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:23.418465 3380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:23.418502 3380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:23.419790 3380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:23.419845 3380 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:56:23.419868 3380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:23.419888 3380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:23.419910 3380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:23.419936 3380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:23.421525 3380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:38215
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:23.421821 3380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:23.422055 3380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:23.427107 3388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:23.427223 3386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:23.427137 3385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:23.428136 3380 server_base.cc:1034] running on GCE node
I20260620 22:56:23.428414 3380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:23.428732 3380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:23.429891 3380 hybrid_clock.cc:648] HybridClock initialized: now 1781996183429867 us; error 40 us; skew 500 ppm
Jun 20 22:56:23 dist-test-slave-rdl9 krb5kdc[3367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:23.432762 3380 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:23.433262 3380 webserver.cc:460] Webserver started at http://127.1.166.62:42317/ using document root <none> and password file <none>
I20260620 22:56:23.433465 3380 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:23.433513 3380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:23.433641 3380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:23.434814 3380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "187a8cbeed6941aeac53e329412146d8"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "fe7b3235f212b6743b677b7c22bf7ed9"
server_key_iv: "b9d083461b6e35e13b2d501bc96b73fc"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.435160 3380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "187a8cbeed6941aeac53e329412146d8"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "fe7b3235f212b6743b677b7c22bf7ed9"
server_key_iv: "b9d083461b6e35e13b2d501bc96b73fc"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.436609 3380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:23.437680 3395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:23.437924 3380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:23.438019 3380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "187a8cbeed6941aeac53e329412146d8"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "fe7b3235f212b6743b677b7c22bf7ed9"
server_key_iv: "b9d083461b6e35e13b2d501bc96b73fc"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.438089 3380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:23.450117 3380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:23.452414 3380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:23.452572 3380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:23.457613 3380 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:38215
I20260620 22:56:23.457693 3457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:38215 every 8 connection(s)
I20260620 22:56:23.458053 3380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:23.458819 3458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:23.461510 3458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: Bootstrap starting.
I20260620 22:56:23.462239 3458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:23.462590 3458 log.cc:824] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:23.463686 3458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: No bootstrap required, opened a new log
I20260620 22:56:23.466538 3458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187a8cbeed6941aeac53e329412146d8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38215 } }
I20260620 22:56:23.466750 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 3380
I20260620 22:56:23.466809 3458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:23.466849 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:23.466850 3458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187a8cbeed6941aeac53e329412146d8, State: Initialized, Role: FOLLOWER
I20260620 22:56:23.467029 1688 external_mini_cluster.cc:1428] Setting key d451181fd8389c5e114d5156089554f3
I20260620 22:56:23.467063 3458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187a8cbeed6941aeac53e329412146d8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38215 } }
I20260620 22:56:23.467177 3458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:23.467221 3458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:23.467276 3458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:23.468428 3458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187a8cbeed6941aeac53e329412146d8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38215 } }
I20260620 22:56:23.468573 3458 leader_election.cc:302] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 187a8cbeed6941aeac53e329412146d8; no voters:
I20260620 22:56:23.468788 3458 leader_election.cc:288] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:23.469002 3463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:23.469245 3463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [term 1 LEADER]: Becoming Leader. State: Replica: 187a8cbeed6941aeac53e329412146d8, State: Running, Role: LEADER
I20260620 22:56:23.469309 3458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:23.469393 3463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187a8cbeed6941aeac53e329412146d8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38215 } }
I20260620 22:56:23.470005 3465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 187a8cbeed6941aeac53e329412146d8. Latest consensus state: current_term: 1 leader_uuid: "187a8cbeed6941aeac53e329412146d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187a8cbeed6941aeac53e329412146d8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38215 } } }
I20260620 22:56:23.470150 3465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:23.470525 3464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "187a8cbeed6941aeac53e329412146d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187a8cbeed6941aeac53e329412146d8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38215 } } }
I20260620 22:56:23.470651 3464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:23.470917 3472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:23.471836 3472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:23.474035 3472 catalog_manager.cc:1347] Generated new cluster ID: c02a0c1d690e42dca47c068c16a63eb0
I20260620 22:56:23.474102 3472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:56:23 dist-test-slave-rdl9 krb5kdc[3367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:23.481884 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.467817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.1.166.62:38215)
0620 22:56:23.468169 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:23.468185 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:23.468236 (+ 51us) server_negotiation.cc:407] Connection header received
0620 22:56:23.468418 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:23.468428 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:23.468521 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:23.468731 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:23.470194 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.471923 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.473066 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.473766 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.473900 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.477144 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:23.477150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:23.477150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:23.477167 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:23.480057 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.480552 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.480555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.480556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.480697 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.480941 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.480942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.480942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.481294 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:23.481469 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:23.481543 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 22:56:23.481693 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":182,"threads_started":1}
I20260620 22:56:23.481971 3472 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:23.482846 3472 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:23.497464 3472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: Generated new TSK 0
I20260620 22:56:23.497671 3472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:23.530598 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38215
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:23.650836 3484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:23.651093 3484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:23.651134 3484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:23.652395 3484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:23.652448 3484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:23.652474 3484 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:56:23.652486 3484 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:56:23.652505 3484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:23.654089 3484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:23.654382 3484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:23.654614 3484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:23.660003 3490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:23.659983 3489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:23.659992 3492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:23.660012 3484 server_base.cc:1034] running on GCE node
I20260620 22:56:23.660430 3484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:23.660742 3484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:23.661931 3484 hybrid_clock.cc:648] HybridClock initialized: now 1781996183661893 us; error 57 us; skew 500 ppm
Jun 20 22:56:23 dist-test-slave-rdl9 krb5kdc[3367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:23.664920 3484 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:23.665496 3484 webserver.cc:460] Webserver started at http://127.1.166.1:44871/ using document root <none> and password file <none>
I20260620 22:56:23.665741 3484 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:23.665799 3484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:23.665949 3484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:23.667143 3484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "13e7fc63ce75476d9a8000c09e1ff47f"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "57a63a168bc913f58d34464e9f441fbf"
server_key_iv: "7493231cb576ec7f946341f8682ed913"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.667675 3484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "13e7fc63ce75476d9a8000c09e1ff47f"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "57a63a168bc913f58d34464e9f441fbf"
server_key_iv: "7493231cb576ec7f946341f8682ed913"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.669236 3484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:56:23.670112 3499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:23.670318 3484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:23.670456 3484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "13e7fc63ce75476d9a8000c09e1ff47f"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "57a63a168bc913f58d34464e9f441fbf"
server_key_iv: "7493231cb576ec7f946341f8682ed913"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.670526 3484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:23.682807 3484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:23.684792 3484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:23.684927 3484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:23.685415 3484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:23.685458 3484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:23.685490 3484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:23.685525 3484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:23.692026 3484 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:44549
I20260620 22:56:23.692171 3614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:44549 every 8 connection(s)
Jun 20 22:56:23 dist-test-slave-rdl9 krb5kdc[3367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:23.692466 3484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:56:23.695453 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.685878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:58903 (local address 127.1.166.62:38215)
0620 22:56:23.685949 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:23.685952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:23.686108 (+ 156us) server_negotiation.cc:407] Connection header received
0620 22:56:23.686642 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:23.686643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:23.686656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:23.686737 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:23.687633 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.688396 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.689881 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.690188 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.690280 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.692722 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:23.692723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:23.692724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:23.692737 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:23.694092 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.694526 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.694527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.694528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.694585 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.694816 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.694817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.694817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.694942 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:23.694996 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:23.695264 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 22:56:23.695350 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:56:23.695633 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.685837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38215 (local address 127.1.166.1:58903)
0620 22:56:23.685982 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:23.686021 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:23.686545 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:23.686802 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:23.686805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:23.687141 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:23.687520 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:23.687528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.688515 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.688516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:23.689782 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:23.689785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.690305 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.690307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:23.690497 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.691072 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:23.691084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:23.692620 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:23.694175 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:23.694183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:23.694185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:23.694464 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:23.694637 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:23.694637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:23.694638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:23.694754 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:23.694997 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:23.695003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:23.695141 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:56:23.695225 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:56:23.695419 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260620 22:56:23.696092 3615 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38215
I20260620 22:56:23.696178 3615 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:23.696352 3615 heartbeater.cc:511] Master 127.1.166.62:38215 requested a full tablet report, sending...
I20260620 22:56:23.696832 3411 ts_manager.cc:194] Registered new tserver with Master: 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549)
I20260620 22:56:23.697444 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 3484
I20260620 22:56:23.697516 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:23.697623 3411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:58903
I20260620 22:56:23.697677 1688 external_mini_cluster.cc:1428] Setting key 7d8c103ca1e339dfa71e6c64b56e3595
I20260620 22:56:23.703505 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.697042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:36113 (local address 127.1.166.62:38215)
0620 22:56:23.697086 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:23.697088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:23.697197 (+ 109us) server_negotiation.cc:407] Connection header received
0620 22:56:23.697245 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:23.697246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:23.697255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:23.697305 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:23.698163 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.698883 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.699614 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.699822 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.699880 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.701025 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:23.701027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:23.701027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:23.701034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:23.702423 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.702767 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.702768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.702769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.702830 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.702990 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.702991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.702992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.703110 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:23.703174 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:23.703307 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 22:56:23.703403 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:56:23.703519 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.696989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38215 (local address 127.1.166.1:36113)
0620 22:56:23.697138 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:23.697146 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:23.697193 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:23.697829 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:23.697830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:23.698031 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:23.698103 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:23.698105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.699003 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.699004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:23.699517 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:23.699518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.699887 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.699888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:23.699967 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.700486 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:23.700490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:23.700925 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:23.702511 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:23.702512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:23.702513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:23.702701 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:23.702883 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:23.702884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:23.702884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:23.702942 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:23.703163 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:23.703164 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:23.703217 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 22:56:23.703279 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:56:23.703403 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:23.723280 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38215
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:23.813894 3624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:23.814121 3624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:23.814152 3624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:23.815490 3624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:23.815555 3624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:23.815578 3624 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:56:23.815589 3624 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:56:23.815608 3624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:23.817236 3624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:23.817492 3624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:23.817727 3624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:23.822571 3630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:23.822582 3632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:23.822579 3629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:23.822770 3624 server_base.cc:1034] running on GCE node
I20260620 22:56:23.822958 3624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:23.823216 3624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:23.824388 3624 hybrid_clock.cc:648] HybridClock initialized: now 1781996183824377 us; error 31 us; skew 500 ppm
Jun 20 22:56:23 dist-test-slave-rdl9 krb5kdc[3367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:23.827486 3624 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:23.827970 3624 webserver.cc:460] Webserver started at http://127.1.166.2:34527/ using document root <none> and password file <none>
I20260620 22:56:23.828171 3624 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:23.828218 3624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:23.828354 3624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:23.829401 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "2ff35ec9148e44528f091fd65f47f25a"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "e0b139af16821b7ec3cd79d719fd0657"
server_key_iv: "6a8831743fad6c07573e8e2da4a4b24f"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.829703 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "2ff35ec9148e44528f091fd65f47f25a"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "e0b139af16821b7ec3cd79d719fd0657"
server_key_iv: "6a8831743fad6c07573e8e2da4a4b24f"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.832965 3624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 22:56:23.834007 3639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:23.834249 3624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:23.834388 3624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "2ff35ec9148e44528f091fd65f47f25a"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "e0b139af16821b7ec3cd79d719fd0657"
server_key_iv: "6a8831743fad6c07573e8e2da4a4b24f"
server_key_version: "encryptionkey@0"
I20260620 22:56:23.834486 3624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:23.858456 3624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:23.860755 3624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:23.860905 3624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:23.861541 3624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:23.861598 3624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:23.861627 3624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:23.861644 3624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:56:23 dist-test-slave-rdl9 krb5kdc[3367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:23.868968 3624 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:42367
I20260620 22:56:23.869290 3754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:42367 every 8 connection(s)
I20260620 22:56:23.869347 3624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:23.872296 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.862114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:46329 (local address 127.1.166.62:38215)
0620 22:56:23.862200 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:23.862204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:23.862506 (+ 302us) server_negotiation.cc:407] Connection header received
0620 22:56:23.863385 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:23.863387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:23.863404 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:23.863467 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:23.864221 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.864716 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.865769 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.866002 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.866061 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.868904 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:23.868906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:23.868907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:23.868917 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:23.870641 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.871183 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.871185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.871185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.871251 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.871495 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.871496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.871497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.871626 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:23.871701 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:23.872075 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 22:56:23.872206 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 22:56:23.872321 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.862080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38215 (local address 127.1.166.2:46329)
0620 22:56:23.862305 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:23.862385 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:23.863208 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:23.863573 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:23.863578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:23.863897 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:23.864108 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:23.864114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.864808 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.864809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:23.865664 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:23.865668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.866090 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.866091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:23.866218 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.866864 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:23.866875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:23.868737 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:23.870772 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:23.870779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:23.870781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:23.871088 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:23.871306 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:23.871307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:23.871307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:23.871412 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:23.871716 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:23.871723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:23.871872 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 22:56:23.871971 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 22:56:23.872124 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":121,"threads_started":1}
I20260620 22:56:23.872953 3755 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38215
I20260620 22:56:23.873070 3755 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:23.873309 3755 heartbeater.cc:511] Master 127.1.166.62:38215 requested a full tablet report, sending...
I20260620 22:56:23.873834 3412 ts_manager.cc:194] Registered new tserver with Master: 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367)
I20260620 22:56:23.874493 3412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:46329
I20260620 22:56:23.879189 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 3624
I20260620 22:56:23.879283 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:23.879454 1688 external_mini_cluster.cc:1428] Setting key ca9b13853ca83154e9e753fd33d72c7d
I20260620 22:56:23.879791 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.874526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:46089 (local address 127.1.166.62:38215)
0620 22:56:23.874574 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:23.874575 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:23.874669 (+ 94us) server_negotiation.cc:407] Connection header received
0620 22:56:23.874738 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:23.874739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:23.874749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:23.874795 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:23.875159 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.875648 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.876332 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.876527 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.876598 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.877440 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:23.877441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:23.877442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:23.877449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:23.878713 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.879007 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.879008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.879009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.879057 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:23.879223 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:23.879223 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:23.879224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:23.879395 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:23.879450 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:23.879593 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 22:56:23.879670 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:56:23.879799 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:23.874452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38215 (local address 127.1.166.2:46089)
0620 22:56:23.874611 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:23.874622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:23.874682 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:23.874805 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:23.874805 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:23.875026 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:23.875091 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:23.875093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.875736 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.875737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:23.876253 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:23.876254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:23.876602 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:23.876603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:23.876677 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:23.877084 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:23.877087 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:23.877364 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:23.878787 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:23.878788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:23.878788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:23.878948 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:23.879108 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:23.879109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:23.879109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:23.879165 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:23.879462 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:23.879463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:23.879510 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 22:56:23.879569 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:56:23.879670 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:23.905306 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38215
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:23.989590 3764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:23.989868 3764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:23.989917 3764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:23.991281 3764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:23.991348 3764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:23.991382 3764 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:56:23.991400 3764 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:56:23.991429 3764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:23.993091 3764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:23.993384 3764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:23.993636 3764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:23.998870 3769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:23.998896 3770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:23.998896 3772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:23.998978 3764 server_base.cc:1034] running on GCE node
I20260620 22:56:23.999444 3764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:23.999738 3764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:24.000880 3764 hybrid_clock.cc:648] HybridClock initialized: now 1781996184000862 us; error 41 us; skew 500 ppm
Jun 20 22:56:23 dist-test-slave-rdl9 krb5kdc[3367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:24.003520 3764 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:24.003998 3764 webserver.cc:460] Webserver started at http://127.1.166.3:33733/ using document root <none> and password file <none>
I20260620 22:56:24.004204 3764 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:24.004263 3764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:24.004546 3764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:24.005972 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "9ec97f2a36274b07acaca2c44ee4cde6"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "575458bc422f5616da8aba4e2f405122"
server_key_iv: "4afa760785e3760c6911396dd5c502d6"
server_key_version: "encryptionkey@0"
I20260620 22:56:24.006425 3764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "9ec97f2a36274b07acaca2c44ee4cde6"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "575458bc422f5616da8aba4e2f405122"
server_key_iv: "4afa760785e3760c6911396dd5c502d6"
server_key_version: "encryptionkey@0"
I20260620 22:56:24.008065 3764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:24.009127 3779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:24.009403 3764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:24.009521 3764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "9ec97f2a36274b07acaca2c44ee4cde6"
format_stamp: "Formatted at 2026-06-20 22:56:23 on dist-test-slave-rdl9"
server_key: "575458bc422f5616da8aba4e2f405122"
server_key_iv: "4afa760785e3760c6911396dd5c502d6"
server_key_version: "encryptionkey@0"
I20260620 22:56:24.009608 3764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:24.036420 3764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:24.039149 3764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:24.039335 3764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:24.040293 3764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:24.040361 3764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:24.040398 3764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:24.040416 3764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:56:24 dist-test-slave-rdl9 krb5kdc[3367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:24.048223 3764 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:37541
I20260620 22:56:24.048300 3894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:37541 every 8 connection(s)
I20260620 22:56:24.048700 3764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:24.050951 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 3764
I20260620 22:56:24.051028 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:24.051183 1688 external_mini_cluster.cc:1428] Setting key 7d7e729668057c3cf0a09064056a7b08
I20260620 22:56:24.051364 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.040534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:44113 (local address 127.1.166.62:38215)
0620 22:56:24.040591 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.040593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.040883 (+ 290us) server_negotiation.cc:407] Connection header received
0620 22:56:24.041481 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.041482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.041494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.041550 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:24.042461 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.042964 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.044083 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.044299 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.044355 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.047899 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:24.047900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:24.047901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:24.047912 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:24.049644 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:24.050152 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:24.050154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:24.050154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:24.050223 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:24.050535 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:24.050536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:24.050536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:24.050677 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:24.050745 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.051170 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.051265 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:56:24.051471 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.040545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38215 (local address 127.1.166.3:44113)
0620 22:56:24.040747 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.040791 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.041377 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.041657 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.041661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.042006 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:24.042366 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.042376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.043060 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.043061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.043992 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.043995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.044384 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.044385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.044516 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.045144 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:24.045155 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:24.047765 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:24.049758 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:24.049767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:24.049770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:24.050060 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:24.050286 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:24.050287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:24.050287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:24.050455 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:24.050770 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:24.050776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:24.050921 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.051015 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.051219 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
I20260620 22:56:24.052006 3895 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38215
I20260620 22:56:24.052109 3895 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:24.052312 3895 heartbeater.cc:511] Master 127.1.166.62:38215 requested a full tablet report, sending...
I20260620 22:56:24.052688 3411 ts_manager.cc:194] Registered new tserver with Master: 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:24.053162 3411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:44113
I20260620 22:56:24.059223 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.053062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38215 (local address 127.1.166.3:51791)
0620 22:56:24.053304 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.053317 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.053389 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.053527 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.053528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.053767 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:24.053844 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.053847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.054513 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.054514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.055038 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.055040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.055681 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.055682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.055755 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.056248 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:24.056252 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:24.056574 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:24.058178 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:24.058180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:24.058180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:24.058421 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:24.058612 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:24.058613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:24.058613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:24.058676 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:24.058900 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:24.058901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:24.058953 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.059016 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.059130 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":155,"threads_started":1}
I20260620 22:56:24.059248 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.053101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:51791 (local address 127.1.166.62:38215)
0620 22:56:24.053168 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.053170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.053372 (+ 202us) server_negotiation.cc:407] Connection header received
0620 22:56:24.053457 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.053457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.053470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.053521 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:24.053924 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.054433 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.055194 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.055569 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.055645 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.056683 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:24.056684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:24.056685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:24.056694 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:24.058061 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:24.058499 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:24.058500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:24.058500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:24.058564 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:24.058720 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:24.058721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:24.058721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:24.058834 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:24.058886 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.059085 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.059166 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:56:24.063078 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:56:24 dist-test-slave-rdl9 krb5kdc[3367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:56:24 dist-test-slave-rdl9 krb5kdc[3367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:24.087289 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.080438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.1.166.62:38215)
0620 22:56:24.080502 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.080504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.080514 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:24.080542 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.080543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.080553 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.080608 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:24.081209 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.081711 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.082519 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.082713 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.082765 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.085140 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:24.085141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:24.085142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:24.085150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:24.086415 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:24.086710 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:24.086718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:24.086718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:24.086767 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:24.086896 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:24.086896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:24.086897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:24.086995 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:24.087065 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.087137 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.087203 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:56:24.088886 3411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:24.090637 3411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:24.097150 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.092969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42367 (local address 127.0.0.1:57906)
0620 22:56:24.093129 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.093174 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.093236 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.093558 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.093560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.093562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.093631 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.093633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.094966 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.094967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.095969 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.095972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.096833 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.096834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.096926 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.096941 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.097011 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.097062 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":88,"thread_start_us":58,"threads_started":1}
I20260620 22:56:24.097182 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.093248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.1.166.2:42367)
0620 22:56:24.093398 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.093403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.093414 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:24.093454 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.093456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.093471 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.093543 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.093692 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.094902 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.096053 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.096751 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.096820 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.096881 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.097037 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.097076 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260620 22:56:24.097405 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.092975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:37541 (local address 127.0.0.1:53684)
0620 22:56:24.093402 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.093412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.093466 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.093613 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.093614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.093616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.093677 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.093679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.095103 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.095103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.096262 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.096265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.097165 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.097166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.097228 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.097232 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.097297 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.097337 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":21,"thread_start_us":100,"threads_started":1}
I20260620 22:56:24.097651 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.093096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.1.166.3:37541)
0620 22:56:24.093469 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.093473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.093485 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:56:24.093526 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.093528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.093542 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.093607 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.093741 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.094898 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.096354 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.097015 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.097078 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.097128 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.097518 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.097560 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":68,"threads_started":1}
I20260620 22:56:24.098069 3689 tablet_service.cc:1467] Processing CreateTablet for tablet 98e55e71f63448ce80565016416942a8 (DEFAULT_TABLE table=test-table [id=af83386999a74dfa80541bd98ac2df7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:24.098240 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 98e55e71f63448ce80565016416942a8 (DEFAULT_TABLE table=test-table [id=af83386999a74dfa80541bd98ac2df7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:24.098389 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.093168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44549 (local address 127.0.0.1:38622)
0620 22:56:24.093682 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.093692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.093736 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.094006 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.094007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.094008 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.094042 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.094043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.094922 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.094923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.096625 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.096628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.097928 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.097929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.098013 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.098017 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.098135 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.098292 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":59,"thread_start_us":82,"threads_started":1}
I20260620 22:56:24.098482 3689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e55e71f63448ce80565016416942a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.098537 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.093286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.1.166.1:44549)
0620 22:56:24.093451 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.093454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.093797 (+ 343us) server_negotiation.cc:407] Connection header received
0620 22:56:24.093838 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.093840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.093856 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.093915 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.094119 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.094849 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.096718 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.097654 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.097714 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.097770 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.098402 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.098448 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260620 22:56:24.098600 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e55e71f63448ce80565016416942a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.100996 3918 tablet_bootstrap.cc:492] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: Bootstrap starting.
I20260620 22:56:24.101455 3919 tablet_bootstrap.cc:492] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Bootstrap starting.
I20260620 22:56:24.101567 3549 tablet_service.cc:1467] Processing CreateTablet for tablet 98e55e71f63448ce80565016416942a8 (DEFAULT_TABLE table=test-table [id=af83386999a74dfa80541bd98ac2df7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:24.101907 3549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e55e71f63448ce80565016416942a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.101948 3918 tablet_bootstrap.cc:654] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.102000 3919 tablet_bootstrap.cc:654] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.102233 3918 log.cc:824] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:24.102269 3919 log.cc:824] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:24.102993 3919 tablet_bootstrap.cc:492] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: No bootstrap required, opened a new log
I20260620 22:56:24.103055 3919 ts_tablet_manager.cc:1397] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:56:24.103207 3918 tablet_bootstrap.cc:492] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: No bootstrap required, opened a new log
I20260620 22:56:24.103276 3918 ts_tablet_manager.cc:1397] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:56:24.104833 3919 raft_consensus.cc:348] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.105006 3919 raft_consensus.cc:374] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.105052 3919 raft_consensus.cc:729] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff35ec9148e44528f091fd65f47f25a, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.105204 3919 consensus_queue.cc:260] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.105492 3919 ts_tablet_manager.cc:1428] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:56:24.105530 3922 tablet_bootstrap.cc:492] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: Bootstrap starting.
I20260620 22:56:24.105652 3755 heartbeater.cc:503] Master 127.1.166.62:38215 was elected leader, sending a full tablet report...
I20260620 22:56:24.105537 3918 raft_consensus.cc:348] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.105976 3918 raft_consensus.cc:374] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.106024 3918 raft_consensus.cc:729] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec97f2a36274b07acaca2c44ee4cde6, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.106243 3922 tablet_bootstrap.cc:654] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.106458 3918 consensus_queue.cc:260] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.106611 3922 log.cc:824] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:24.106742 3918 ts_tablet_manager.cc:1428] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:56:24.106791 3895 heartbeater.cc:503] Master 127.1.166.62:38215 was elected leader, sending a full tablet report...
I20260620 22:56:24.107867 3922 tablet_bootstrap.cc:492] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: No bootstrap required, opened a new log
I20260620 22:56:24.107956 3922 ts_tablet_manager.cc:1397] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:56:24.111389 3922 raft_consensus.cc:348] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.111697 3922 raft_consensus.cc:374] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.111743 3922 raft_consensus.cc:729] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e7fc63ce75476d9a8000c09e1ff47f, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.111914 3922 consensus_queue.cc:260] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.112293 3922 ts_tablet_manager.cc:1428] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260620 22:56:24.112409 3615 heartbeater.cc:503] Master 127.1.166.62:38215 was elected leader, sending a full tablet report...
I20260620 22:56:24.117690 3924 raft_consensus.cc:482] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:24.117828 3924 raft_consensus.cc:504] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.118192 3924 leader_election.cc:288] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367), 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549)
W20260620 22:56:24.119920 3756 tablet.cc:2369] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:24.122015 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.118496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:60249 (local address 127.1.166.2:42367)
0620 22:56:24.118557 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.118559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.118571 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:56:24.118709 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.118710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.118723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.118787 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.118946 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.119728 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.121136 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.121643 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.121723 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.121736 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.121906 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.121949 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:56:24.122033 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.118450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42367 (local address 127.1.166.3:60249)
0620 22:56:24.118492 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.118504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.118566 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.118801 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.118802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.118804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.118881 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.118884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.119821 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.119823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.121059 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.121062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.121743 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.121744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.121830 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.121834 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.121892 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.121941 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:56:24.122259 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.118566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44549 (local address 127.1.166.3:52579)
0620 22:56:24.118699 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.118709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.118761 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.118890 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.118890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.118892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.118953 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.118955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.120174 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.120175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.121216 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.121217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.121933 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.121934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.122004 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.122008 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.122060 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.122179 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260620 22:56:24.122391 3709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e55e71f63448ce80565016416942a8" candidate_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff35ec9148e44528f091fd65f47f25a" is_pre_election: true
I20260620 22:56:24.122529 3709 raft_consensus.cc:2454] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ec97f2a36274b07acaca2c44ee4cde6 in term 0.
I20260620 22:56:24.122560 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.118647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:52579 (local address 127.1.166.1:44549)
0620 22:56:24.118693 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.118695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.118752 (+ 57us) server_negotiation.cc:407] Connection header received
0620 22:56:24.118811 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.118812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.118824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.118887 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.119009 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.119720 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.121290 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.121852 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.121914 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.121923 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.122439 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.122486 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:24.122740 3783 leader_election.cc:302] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff35ec9148e44528f091fd65f47f25a, 9ec97f2a36274b07acaca2c44ee4cde6; no voters:
I20260620 22:56:24.122867 3924 raft_consensus.cc:2781] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:24.122843 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e55e71f63448ce80565016416942a8" candidate_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" is_pre_election: true
I20260620 22:56:24.122906 3924 raft_consensus.cc:482] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:24.122928 3924 raft_consensus.cc:3037] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.122977 3569 raft_consensus.cc:2454] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ec97f2a36274b07acaca2c44ee4cde6 in term 0.
I20260620 22:56:24.123608 3924 raft_consensus.cc:504] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.123732 3924 leader_election.cc:288] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [CANDIDATE]: Term 1 election: Requested vote from peers 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367), 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549)
I20260620 22:56:24.123917 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e55e71f63448ce80565016416942a8" candidate_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e7fc63ce75476d9a8000c09e1ff47f"
I20260620 22:56:24.123922 3709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e55e71f63448ce80565016416942a8" candidate_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff35ec9148e44528f091fd65f47f25a"
I20260620 22:56:24.123994 3709 raft_consensus.cc:3037] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.123994 3569 raft_consensus.cc:3037] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.124809 3569 raft_consensus.cc:2454] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec97f2a36274b07acaca2c44ee4cde6 in term 1.
I20260620 22:56:24.124809 3709 raft_consensus.cc:2454] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec97f2a36274b07acaca2c44ee4cde6 in term 1.
I20260620 22:56:24.124975 3783 leader_election.cc:302] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff35ec9148e44528f091fd65f47f25a, 9ec97f2a36274b07acaca2c44ee4cde6; no voters:
I20260620 22:56:24.125057 3924 raft_consensus.cc:2781] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:24.125211 3924 raft_consensus.cc:686] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec97f2a36274b07acaca2c44ee4cde6, State: Running, Role: LEADER
I20260620 22:56:24.125299 3924 consensus_queue.cc:237] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.126098 3411 catalog_manager.cc:5495] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3). New cstate: current_term: 1 leader_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } health_report { overall_health: HEALTHY } } }
Jun 20 22:56:24 dist-test-slave-rdl9 krb5kdc[3367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996183, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:24.158363 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.151193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.1.166.62:38215)
0620 22:56:24.151263 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.151265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.151351 (+ 86us) server_negotiation.cc:407] Connection header received
0620 22:56:24.151429 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.151430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.151440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.151479 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:24.151952 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.152436 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.153115 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.153291 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.153344 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.155576 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:24.155577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:24.155578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:24.155586 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:24.157209 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:24.157619 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:24.157620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:24.157621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:24.157680 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:24.157870 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:24.157871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:24.157872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:24.158008 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:24.158074 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.158192 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.158267 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:56:24.158470 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.151106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38215 (local address 127.0.0.1:40484)
0620 22:56:24.151264 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.151274 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.151363 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.151489 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.151490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.151820 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:24.151879 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.151882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.152506 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.152506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.153046 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.153048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.153347 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.153347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.153392 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.153891 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:24.153897 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:24.155485 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:24.157303 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:24.157315 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:24.157318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:24.157556 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:24.157746 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:24.157747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:24.157747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:24.157802 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:24.158070 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:24.158076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:24.158121 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.158168 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.158322 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":55,"thread_start_us":93,"threads_started":1}
I20260620 22:56:24.160612 3411 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:40484:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:56:24.160835 3411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:24.162510 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 615adeb9b41b496991000d48b8f026fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=576d895251a140f1810be5e9d0a55dcb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:56:24.162546 3549 tablet_service.cc:1467] Processing CreateTablet for tablet 615adeb9b41b496991000d48b8f026fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=576d895251a140f1810be5e9d0a55dcb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:56:24.162573 3689 tablet_service.cc:1467] Processing CreateTablet for tablet 615adeb9b41b496991000d48b8f026fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=576d895251a140f1810be5e9d0a55dcb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:56:24.162693 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615adeb9b41b496991000d48b8f026fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.162693 3549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615adeb9b41b496991000d48b8f026fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.162732 3689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615adeb9b41b496991000d48b8f026fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.164103 3919 tablet_bootstrap.cc:492] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: Bootstrap starting.
I20260620 22:56:24.164507 3918 tablet_bootstrap.cc:492] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: Bootstrap starting.
I20260620 22:56:24.164534 3922 tablet_bootstrap.cc:492] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: Bootstrap starting.
I20260620 22:56:24.164819 3919 tablet_bootstrap.cc:654] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.164951 3918 tablet_bootstrap.cc:654] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.165197 3922 tablet_bootstrap.cc:654] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.165621 3918 tablet_bootstrap.cc:492] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: No bootstrap required, opened a new log
I20260620 22:56:24.165676 3918 ts_tablet_manager.cc:1397] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:24.165803 3922 tablet_bootstrap.cc:492] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: No bootstrap required, opened a new log
I20260620 22:56:24.165830 3919 tablet_bootstrap.cc:492] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: No bootstrap required, opened a new log
I20260620 22:56:24.165858 3922 ts_tablet_manager.cc:1397] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:56:24.165829 3918 raft_consensus.cc:348] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.165879 3919 ts_tablet_manager.cc:1397] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:56:24.165900 3918 raft_consensus.cc:374] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.165930 3918 raft_consensus.cc:729] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec97f2a36274b07acaca2c44ee4cde6, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.165969 3918 consensus_queue.cc:260] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.165999 3922 raft_consensus.cc:348] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.166062 3922 raft_consensus.cc:374] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.166059 3918 ts_tablet_manager.cc:1428] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:24.166085 3922 raft_consensus.cc:729] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e7fc63ce75476d9a8000c09e1ff47f, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.166034 3919 raft_consensus.cc:348] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.166138 3919 raft_consensus.cc:374] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.166131 3922 consensus_queue.cc:260] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.166172 3919 raft_consensus.cc:729] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff35ec9148e44528f091fd65f47f25a, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.166208 3922 ts_tablet_manager.cc:1428] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:24.166186 3924 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.166224 3919 consensus_queue.cc:260] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.166245 3924 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:24.166366 3919 ts_tablet_manager.cc:1428] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:24.166292 3926 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.166425 3926 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:24.166416 3923 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.166508 3923 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:24.169559 3926 raft_consensus.cc:482] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:24.169664 3926 raft_consensus.cc:504] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.170024 3926 leader_election.cc:288] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367), 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:24.174142 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.170259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:46915 (local address 127.1.166.3:37541)
0620 22:56:24.170330 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.170332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.170366 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:56:24.170392 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.170392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.170403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.170456 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.170617 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.171365 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.172789 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.173701 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.173791 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.173806 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.174005 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.174064 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:56:24.174142 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.170393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:45043 (local address 127.1.166.2:42367)
0620 22:56:24.170445 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.170447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.170457 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:24.170523 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.170524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.170532 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.170574 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.170711 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.171501 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.172855 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.173706 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.173791 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.173806 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.174005 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.174066 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:56:24.174166 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.170183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:37541 (local address 127.1.166.1:46915)
0620 22:56:24.170236 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.170247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.170285 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.170469 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.170470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.170472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.170548 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.170550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.171468 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.171469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.172676 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.172680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.173810 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.173812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.173931 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.173934 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.173994 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.174038 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":27}
I20260620 22:56:24.174177 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.170201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42367 (local address 127.1.166.1:45043)
0620 22:56:24.170405 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.170416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.170469 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.170578 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.170579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.170581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.170653 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.170654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.171660 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.171662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.172745 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.172747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.173828 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.173829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.173910 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.173917 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.173982 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.174038 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260620 22:56:24.174590 3709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615adeb9b41b496991000d48b8f026fa" candidate_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff35ec9148e44528f091fd65f47f25a" is_pre_election: true
I20260620 22:56:24.174600 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615adeb9b41b496991000d48b8f026fa" candidate_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" is_pre_election: true
I20260620 22:56:24.174728 3709 raft_consensus.cc:2454] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e7fc63ce75476d9a8000c09e1ff47f in term 0.
I20260620 22:56:24.174733 3849 raft_consensus.cc:2454] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e7fc63ce75476d9a8000c09e1ff47f in term 0.
I20260620 22:56:24.174932 3503 leader_election.cc:302] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e7fc63ce75476d9a8000c09e1ff47f, 2ff35ec9148e44528f091fd65f47f25a; no voters:
I20260620 22:56:24.175143 3926 raft_consensus.cc:2781] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:24.175187 3926 raft_consensus.cc:482] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:24.175202 3926 raft_consensus.cc:3037] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.175957 3926 raft_consensus.cc:504] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.176119 3926 leader_election.cc:288] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [CANDIDATE]: Term 1 election: Requested vote from peers 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367), 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:24.176363 3709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615adeb9b41b496991000d48b8f026fa" candidate_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff35ec9148e44528f091fd65f47f25a"
I20260620 22:56:24.176452 3709 raft_consensus.cc:3037] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.177109 3709 raft_consensus.cc:2454] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e7fc63ce75476d9a8000c09e1ff47f in term 1.
I20260620 22:56:24.177242 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615adeb9b41b496991000d48b8f026fa" candidate_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec97f2a36274b07acaca2c44ee4cde6"
I20260620 22:56:24.177331 3849 raft_consensus.cc:3037] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.177438 3503 leader_election.cc:302] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e7fc63ce75476d9a8000c09e1ff47f, 2ff35ec9148e44528f091fd65f47f25a; no voters:
I20260620 22:56:24.177614 3926 raft_consensus.cc:2781] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:24.177802 3926 raft_consensus.cc:686] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 LEADER]: Becoming Leader. State: Replica: 13e7fc63ce75476d9a8000c09e1ff47f, State: Running, Role: LEADER
I20260620 22:56:24.177942 3926 consensus_queue.cc:237] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.178135 3849 raft_consensus.cc:2454] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e7fc63ce75476d9a8000c09e1ff47f in term 1.
I20260620 22:56:24.178881 3940 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: TxnStatusTablet state changed. Reason: New leader 13e7fc63ce75476d9a8000c09e1ff47f. Latest consensus state: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.179095 3940 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:56:24.179101 3410 catalog_manager.cc:5495] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1). New cstate: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20260620 22:56:24.179355 3942 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:56:24.179488 3942 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:56:24.188906 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.185607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.1.166.1:44549)
0620 22:56:24.185691 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.185694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.185705 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:24.185838 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.185839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.185865 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.185921 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:24.186097 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.186753 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.187917 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.188159 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.188236 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.188434 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.188635 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.188736 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.188793 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.188822 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 22:56:24.188930 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.185342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44549 (local address 127.0.0.1:38624)
0620 22:56:24.185606 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.185630 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.185720 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.185934 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.185936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.185937 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:56:24.186028 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.186039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.186841 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.186843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.187725 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.187734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.188245 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.188246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.188324 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.188347 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.188730 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.188742 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.188790 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.188862 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 22:56:24.191237 3849 raft_consensus.cc:1261] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Refusing update from remote peer 13e7fc63ce75476d9a8000c09e1ff47f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:24.191237 3709 raft_consensus.cc:1261] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Refusing update from remote peer 13e7fc63ce75476d9a8000c09e1ff47f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:24.191629 3940 consensus_queue.cc:1035] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:24.191776 3926 consensus_queue.cc:1035] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:24.193056 3924 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: TxnStatusTablet state changed. Reason: New leader 13e7fc63ce75476d9a8000c09e1ff47f. Latest consensus state: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.193094 3923 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: TxnStatusTablet state changed. Reason: New leader 13e7fc63ce75476d9a8000c09e1ff47f. Latest consensus state: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.193187 3923 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:24.193183 3924 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 22:56:24.193259 3616 tablet.cc:2369] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:24.195878 3926 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.196003 3926 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:56:24.196076 3945 mvcc.cc:205] Tried to move back new op lower bound from 7299056370443902976 to 7299056370393554944. Current Snapshot: MvccSnapshot[applied={T|T < 7299056370443902976}]
I20260620 22:56:24.196388 3926 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.196467 3926 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 13e7fc63ce75476d9a8000c09e1ff47f: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:56:24.197095 3924 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.197204 3924 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 9ec97f2a36274b07acaca2c44ee4cde6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:24.197506 3947 mvcc.cc:205] Tried to move back new op lower bound from 7299056370443902976 to 7299056370393554944. Current Snapshot: MvccSnapshot[applied={T|T < 7299056370443902976 or (T in {7299056370443902976})}]
I20260620 22:56:24.198091 3923 tablet_replica.cc:439] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } } }
I20260620 22:56:24.198230 3923 tablet_replica.cc:442] T 615adeb9b41b496991000d48b8f026fa P 2ff35ec9148e44528f091fd65f47f25a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:56:24.198252 3946 mvcc.cc:205] Tried to move back new op lower bound from 7299056370443902976 to 7299056370393554944. Current Snapshot: MvccSnapshot[applied={T|T < 7299056370443902976}]
I20260620 22:56:24.202044 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.199394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.1.166.3:37541)
0620 22:56:24.199456 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.199458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.199468 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:24.199506 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.199507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.199529 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.199604 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:24.199781 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.200338 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.201184 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.201413 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.201479 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.201651 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.201792 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.201871 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.201914 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.201949 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:56:24.208096 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.205234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:58563 (local address 127.1.166.1:44549)
0620 22:56:24.205284 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.205286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.205293 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:24.205322 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.205323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.205335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.205383 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:24.205552 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.206090 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.207270 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.207471 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.207551 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.207830 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.207875 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.207933 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.208003 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.208024 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:56:24.208145 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.205117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44549 (local address 127.1.166.3:58563)
0620 22:56:24.205193 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.205209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.205266 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.205401 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.205401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.205403 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:56:24.205480 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.205482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.206181 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.206182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.207150 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.207153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.207565 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.207565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.207646 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.207690 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.207947 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.207949 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.207999 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.208054 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:56:24.211021 3709 raft_consensus.cc:1261] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Refusing update from remote peer 9ec97f2a36274b07acaca2c44ee4cde6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:24.211113 3569 raft_consensus.cc:1261] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Refusing update from remote peer 9ec97f2a36274b07acaca2c44ee4cde6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:24.211473 3928 consensus_queue.cc:1035] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:24.211625 3924 consensus_queue.cc:1035] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:24.221021 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.218321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:37541 (local address 127.1.166.1:58295)
0620 22:56:24.218535 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.218546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.218592 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.218699 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.218699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.218701 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:56:24.218765 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.218766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.219390 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.219391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.220162 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.220165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.220559 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.220560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.220627 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.220643 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.220878 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.220879 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.220908 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.220947 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":118,"threads_started":1}
I20260620 22:56:24.221024 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.218548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:58295 (local address 127.1.166.3:37541)
0620 22:56:24.218604 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.218606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.218614 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:24.218636 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.218637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.218649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.218689 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:24.218822 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.219319 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.220232 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.220498 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.220551 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.220777 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.220816 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.220876 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.220924 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.220946 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:56:24.230962 3410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:24.231094 3410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:24.231894 3410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: Sending DeleteTablet for 3 replicas of tablet 98e55e71f63448ce80565016416942a8
I20260620 22:56:24.232237 3829 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e55e71f63448ce80565016416942a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:24 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:53684
I20260620 22:56:24.232409 3689 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e55e71f63448ce80565016416942a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:24 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:57906
I20260620 22:56:24.232606 3549 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e55e71f63448ce80565016416942a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:24 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38622
I20260620 22:56:24.232832 3967 tablet_replica.cc:330] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: stopping tablet replica
I20260620 22:56:24.232985 3967 raft_consensus.cc:2229] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:24.233107 3967 raft_consensus.cc:2258] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:24.233105 3410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:24.233274 3410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:24.233714 3966 tablet_replica.cc:330] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: stopping tablet replica
I20260620 22:56:24.234375 3967 ts_tablet_manager.cc:1905] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:24.234936 3829 tablet_service.cc:1467] Processing CreateTablet for tablet add249035f29460dbcb4401cd9d5dc4b (DEFAULT_TABLE table=test-table [id=4233cccf37a9495982af115dbf0c3062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:24.235059 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add249035f29460dbcb4401cd9d5dc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.236377 3918 tablet_bootstrap.cc:492] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: Bootstrap starting.
I20260620 22:56:24.236411 3967 ts_tablet_manager.cc:1918] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:24.236599 3967 log.cc:1196] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/98e55e71f63448ce80565016416942a8
I20260620 22:56:24.236788 3918 tablet_bootstrap.cc:654] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.236979 3967 ts_tablet_manager.cc:1939] T 98e55e71f63448ce80565016416942a8 P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting consensus metadata
I20260620 22:56:24.237496 3399 catalog_manager.cc:4841] TS 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549): tablet 98e55e71f63448ce80565016416942a8 (table test-table [id=af83386999a74dfa80541bd98ac2df7f]) successfully deleted
I20260620 22:56:24.237560 3549 tablet_service.cc:1467] Processing CreateTablet for tablet add249035f29460dbcb4401cd9d5dc4b (DEFAULT_TABLE table=test-table [id=4233cccf37a9495982af115dbf0c3062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:24.237645 3549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add249035f29460dbcb4401cd9d5dc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.237731 3689 tablet_service.cc:1467] Processing CreateTablet for tablet add249035f29460dbcb4401cd9d5dc4b (DEFAULT_TABLE table=test-table [id=4233cccf37a9495982af115dbf0c3062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:24.237825 3689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add249035f29460dbcb4401cd9d5dc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:24.238803 3922 tablet_bootstrap.cc:492] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: Bootstrap starting.
I20260620 22:56:24.239120 3919 tablet_bootstrap.cc:492] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: Bootstrap starting.
I20260620 22:56:24.239203 3922 tablet_bootstrap.cc:654] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.239502 3966 raft_consensus.cc:2229] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:24.239522 3919 tablet_bootstrap.cc:654] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:24.239570 3966 raft_consensus.cc:2258] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:24.239801 3965 tablet_replica.cc:330] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: stopping tablet replica
I20260620 22:56:24.239908 3965 raft_consensus.cc:2229] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:56:24.240005 3922 tablet_bootstrap.cc:492] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: No bootstrap required, opened a new log
I20260620 22:56:24.240005 3965 raft_consensus.cc:2258] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:24.240039 3922 ts_tablet_manager.cc:1397] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:56:24.240178 3922 raft_consensus.cc:348] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.240245 3922 raft_consensus.cc:374] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.240265 3922 raft_consensus.cc:729] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e7fc63ce75476d9a8000c09e1ff47f, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.240307 3922 consensus_queue.cc:260] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.240396 3922 ts_tablet_manager.cc:1428] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:24.240494 3966 ts_tablet_manager.cc:1905] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:24.240938 3918 tablet_bootstrap.cc:492] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: No bootstrap required, opened a new log
I20260620 22:56:24.240984 3918 ts_tablet_manager.cc:1397] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 22:56:24.241108 3918 raft_consensus.cc:348] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.241183 3918 raft_consensus.cc:374] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.241201 3918 raft_consensus.cc:729] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec97f2a36274b07acaca2c44ee4cde6, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.241238 3918 consensus_queue.cc:260] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.241312 3918 ts_tablet_manager.cc:1428] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:24.242362 3966 ts_tablet_manager.cc:1918] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:24.242472 3966 log.cc:1196] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/98e55e71f63448ce80565016416942a8
I20260620 22:56:24.242547 3965 ts_tablet_manager.cc:1905] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:24.242784 3966 ts_tablet_manager.cc:1939] T 98e55e71f63448ce80565016416942a8 P 2ff35ec9148e44528f091fd65f47f25a: Deleting consensus metadata
I20260620 22:56:24.243206 3399 catalog_manager.cc:4841] TS 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367): tablet 98e55e71f63448ce80565016416942a8 (table test-table [id=af83386999a74dfa80541bd98ac2df7f]) successfully deleted
I20260620 22:56:24.244287 3965 ts_tablet_manager.cc:1918] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:24.244370 3965 log.cc:1196] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/98e55e71f63448ce80565016416942a8
I20260620 22:56:24.244583 3965 ts_tablet_manager.cc:1939] T 98e55e71f63448ce80565016416942a8 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting consensus metadata
I20260620 22:56:24.244949 3397 catalog_manager.cc:4841] TS 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541): tablet 98e55e71f63448ce80565016416942a8 (table test-table [id=af83386999a74dfa80541bd98ac2df7f]) successfully deleted
I20260620 22:56:24.246750 3919 tablet_bootstrap.cc:492] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: No bootstrap required, opened a new log
I20260620 22:56:24.246799 3919 ts_tablet_manager.cc:1397] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 22:56:24.246958 3919 raft_consensus.cc:348] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.247022 3919 raft_consensus.cc:374] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:24.247040 3919 raft_consensus.cc:729] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff35ec9148e44528f091fd65f47f25a, State: Initialized, Role: FOLLOWER
I20260620 22:56:24.247079 3919 consensus_queue.cc:260] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.247166 3919 ts_tablet_manager.cc:1428] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260620 22:56:24.300329 3896 tablet.cc:2369] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:24.374740 3923 raft_consensus.cc:482] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:24.374862 3923 raft_consensus.cc:504] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.375216 3923 leader_election.cc:288] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549), 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:24.378643 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.375544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:41331 (local address 127.1.166.3:37541)
0620 22:56:24.375592 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.375595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.375815 (+ 220us) server_negotiation.cc:407] Connection header received
0620 22:56:24.375879 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.375880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.375887 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.375937 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.376110 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.376784 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.377848 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.378266 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.378362 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.378374 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.378524 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.378562 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:24.378660 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.375446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:37541 (local address 127.1.166.2:41331)
0620 22:56:24.375719 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.375731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.375813 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.375957 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.375958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.375960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.376034 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.376035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.376868 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.376869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.377740 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.377743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.378377 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.378378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.378450 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.378456 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.378514 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.378563 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":36,"thread_start_us":62,"threads_started":1}
I20260620 22:56:24.379016 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add249035f29460dbcb4401cd9d5dc4b" candidate_uuid: "2ff35ec9148e44528f091fd65f47f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" is_pre_election: true
I20260620 22:56:24.379120 3849 raft_consensus.cc:2454] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ff35ec9148e44528f091fd65f47f25a in term 0.
I20260620 22:56:24.379221 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.375635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:45559 (local address 127.1.166.1:44549)
0620 22:56:24.375719 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.375721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.375729 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:24.375756 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.375757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.375764 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.375808 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:24.375993 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.376741 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.378218 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.378749 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.378816 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.378829 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.379105 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.379151 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 22:56:24.379514 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.375446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44549 (local address 127.1.166.2:45559)
0620 22:56:24.375587 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.375598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.375656 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.375858 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.375860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.375862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:24.375939 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.375941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.376842 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.376843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.378111 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.378113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.378918 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.378920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.378994 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.378998 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.379339 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.379425 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":31,"thread_start_us":64,"threads_started":1}
I20260620 22:56:24.379302 3641 leader_election.cc:302] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff35ec9148e44528f091fd65f47f25a, 9ec97f2a36274b07acaca2c44ee4cde6; no voters:
I20260620 22:56:24.379668 3923 raft_consensus.cc:2781] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:24.379712 3923 raft_consensus.cc:482] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:24.379733 3923 raft_consensus.cc:3037] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.379746 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add249035f29460dbcb4401cd9d5dc4b" candidate_uuid: "2ff35ec9148e44528f091fd65f47f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" is_pre_election: true
I20260620 22:56:24.379844 3569 raft_consensus.cc:2454] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ff35ec9148e44528f091fd65f47f25a in term 0.
I20260620 22:56:24.380581 3923 raft_consensus.cc:504] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.380724 3923 leader_election.cc:288] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [CANDIDATE]: Term 1 election: Requested vote from peers 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549), 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:24.380882 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add249035f29460dbcb4401cd9d5dc4b" candidate_uuid: "2ff35ec9148e44528f091fd65f47f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e7fc63ce75476d9a8000c09e1ff47f"
I20260620 22:56:24.380962 3569 raft_consensus.cc:3037] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.380954 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add249035f29460dbcb4401cd9d5dc4b" candidate_uuid: "2ff35ec9148e44528f091fd65f47f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec97f2a36274b07acaca2c44ee4cde6"
I20260620 22:56:24.381043 3849 raft_consensus.cc:3037] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:24.381713 3849 raft_consensus.cc:2454] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ff35ec9148e44528f091fd65f47f25a in term 1.
I20260620 22:56:24.381842 3569 raft_consensus.cc:2454] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ff35ec9148e44528f091fd65f47f25a in term 1.
I20260620 22:56:24.381876 3641 leader_election.cc:302] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff35ec9148e44528f091fd65f47f25a, 9ec97f2a36274b07acaca2c44ee4cde6; no voters:
I20260620 22:56:24.381968 3923 raft_consensus.cc:2781] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:24.382077 3923 raft_consensus.cc:686] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 1 LEADER]: Becoming Leader. State: Replica: 2ff35ec9148e44528f091fd65f47f25a, State: Running, Role: LEADER
I20260620 22:56:24.382159 3923 consensus_queue.cc:237] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:24.382913 3410 catalog_manager.cc:5495] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2). New cstate: current_term: 1 leader_uuid: "2ff35ec9148e44528f091fd65f47f25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20260620 22:56:24.389600 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.387404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.1.166.2:42367)
0620 22:56:24.387457 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.387458 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.387465 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:24.387500 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.387500 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.387522 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.387579 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:24.387689 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.388190 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.388872 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.389069 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.389126 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.389267 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.389379 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.389444 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.389490 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.389516 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:56:24.394932 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.392286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:60347 (local address 127.1.166.1:44549)
0620 22:56:24.392336 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.392338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.392400 (+ 62us) server_negotiation.cc:407] Connection header received
0620 22:56:24.392595 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.392597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.392609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.392679 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:24.392814 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.393288 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.394172 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.394448 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.394511 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.394733 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.394772 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.394801 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.394847 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.394865 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:56:24.395175 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.392191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44549 (local address 127.1.166.2:60347)
0620 22:56:24.392348 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.392359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.392413 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.392685 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.392687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.392689 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:56:24.392763 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.392764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.393380 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.393381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.394057 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.394061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.394543 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.394544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.394612 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.394651 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.394811 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.394812 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.394846 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.394888 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260620 22:56:24.397531 3849 raft_consensus.cc:1261] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Refusing update from remote peer 2ff35ec9148e44528f091fd65f47f25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:24.397511 3569 raft_consensus.cc:1261] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Refusing update from remote peer 2ff35ec9148e44528f091fd65f47f25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:24.397918 3973 consensus_queue.cc:1035] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:24.398083 3923 consensus_queue.cc:1035] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:24.407282 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.403679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:40023 (local address 127.1.166.2:42367)
0620 22:56:24.403733 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:24.403736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:24.404868 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 22:56:24.404924 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:24.404925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:24.404935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:24.404985 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:24.405065 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.405724 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.406615 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.406835 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.406898 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.407080 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.407106 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.407137 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:24.407185 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 22:56:24.407210 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:56:24.407302 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:24.403543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42367 (local address 127.1.166.1:40023)
0620 22:56:24.404804 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:24.404817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:24.404876 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:24.404986 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:24.404986 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:24.404987 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:56:24.405031 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.405032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.405801 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.405802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.406535 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:24.406539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:24.406938 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:24.406939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:24.406998 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:24.407007 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:56:24.407140 (+ 133us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:56:24.407141 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:56:24.407182 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 22:56:24.407234 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222}
W20260620 22:56:24.407718 3946 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 22:56:25.932883 3410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:25.933019 3410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:25.934146 3410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: Sending DeleteTablet for 3 replicas of tablet add249035f29460dbcb4401cd9d5dc4b
I20260620 22:56:25.934552 3549 tablet_service.cc:1514] Processing DeleteTablet for tablet add249035f29460dbcb4401cd9d5dc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:25 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38622
I20260620 22:56:25.934758 3829 tablet_service.cc:1514] Processing DeleteTablet for tablet add249035f29460dbcb4401cd9d5dc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:25 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:53684
I20260620 22:56:25.934965 3689 tablet_service.cc:1514] Processing DeleteTablet for tablet add249035f29460dbcb4401cd9d5dc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:25 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:57906
I20260620 22:56:25.935170 4004 tablet_replica.cc:330] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: stopping tablet replica
I20260620 22:56:25.935247 4005 tablet_replica.cc:330] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: stopping tablet replica
I20260620 22:56:25.935299 4005 raft_consensus.cc:2229] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:56:25.935374 4004 raft_consensus.cc:2229] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:25.935437 4004 raft_consensus.cc:2258] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:25.935588 4005 raft_consensus.cc:2258] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:25.935874 4004 ts_tablet_manager.cc:1905] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:25.936659 4005 ts_tablet_manager.cc:1905] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:25.937182 4003 tablet_replica.cc:330] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: stopping tablet replica
I20260620 22:56:25.937260 4003 raft_consensus.cc:2229] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:25.937307 4003 raft_consensus.cc:2258] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:25.937556 4003 ts_tablet_manager.cc:1905] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:25.938267 4005 ts_tablet_manager.cc:1918] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:56:25.938372 4005 log.cc:1196] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/add249035f29460dbcb4401cd9d5dc4b
I20260620 22:56:25.938669 4005 ts_tablet_manager.cc:1939] T add249035f29460dbcb4401cd9d5dc4b P 2ff35ec9148e44528f091fd65f47f25a: Deleting consensus metadata
I20260620 22:56:25.939013 3399 catalog_manager.cc:4841] TS 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367): tablet add249035f29460dbcb4401cd9d5dc4b (table test-table [id=4233cccf37a9495982af115dbf0c3062]) successfully deleted
I20260620 22:56:25.939131 4003 ts_tablet_manager.cc:1918] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:56:25.939183 4003 log.cc:1196] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/add249035f29460dbcb4401cd9d5dc4b
I20260620 22:56:25.939345 4004 ts_tablet_manager.cc:1918] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:56:25.939370 4003 ts_tablet_manager.cc:1939] T add249035f29460dbcb4401cd9d5dc4b P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting consensus metadata
I20260620 22:56:25.939402 4004 log.cc:1196] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/add249035f29460dbcb4401cd9d5dc4b
I20260620 22:56:25.939571 4004 ts_tablet_manager.cc:1939] T add249035f29460dbcb4401cd9d5dc4b P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting consensus metadata
I20260620 22:56:25.939689 3399 catalog_manager.cc:4841] TS 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549): tablet add249035f29460dbcb4401cd9d5dc4b (table test-table [id=4233cccf37a9495982af115dbf0c3062]) successfully deleted
I20260620 22:56:25.939845 3397 catalog_manager.cc:4841] TS 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541): tablet add249035f29460dbcb4401cd9d5dc4b (table test-table [id=4233cccf37a9495982af115dbf0c3062]) successfully deleted
Jun 20 22:56:25 dist-test-slave-rdl9 krb5kdc[3367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:56:25 dist-test-slave-rdl9 krb5kdc[3367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:25.967471 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:25.953925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.1.166.62:38215)
0620 22:56:25.954084 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:25.954085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:25.954095 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:25.954237 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:25.954238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:25.954249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:25.954310 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:25.954945 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:25.955478 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:25.956254 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:25.956460 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:25.956524 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:25.964062 (+ 7538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:25.964064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:25.964065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:25.964074 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:25.965867 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:25.966547 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:25.966548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:25.966548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:25.966628 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:25.966919 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:25.966919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:25.966919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:25.967038 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:25.967129 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:25.967289 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 22:56:25.967389 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":110,"threads_started":1}
I20260620 22:56:25.969553 3410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:25.969765 3410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:25.971161 3549 tablet_service.cc:1467] Processing CreateTablet for tablet 14eda3fe0a764801bde66df2d167fa21 (DEFAULT_TABLE table=test-table [id=497aa9282b91474dbc53c145be6fa32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:25.971160 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 14eda3fe0a764801bde66df2d167fa21 (DEFAULT_TABLE table=test-table [id=497aa9282b91474dbc53c145be6fa32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:25.971186 3689 tablet_service.cc:1467] Processing CreateTablet for tablet 14eda3fe0a764801bde66df2d167fa21 (DEFAULT_TABLE table=test-table [id=497aa9282b91474dbc53c145be6fa32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:25.971315 3689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14eda3fe0a764801bde66df2d167fa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:25.971318 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14eda3fe0a764801bde66df2d167fa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:25.971323 3549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14eda3fe0a764801bde66df2d167fa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:25.973280 4016 tablet_bootstrap.cc:492] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: Bootstrap starting.
I20260620 22:56:25.973382 4015 tablet_bootstrap.cc:492] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: Bootstrap starting.
I20260620 22:56:25.973719 4016 tablet_bootstrap.cc:654] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:25.973739 4015 tablet_bootstrap.cc:654] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:25.974067 4014 tablet_bootstrap.cc:492] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: Bootstrap starting.
I20260620 22:56:25.974298 4015 tablet_bootstrap.cc:492] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: No bootstrap required, opened a new log
I20260620 22:56:25.974331 4016 tablet_bootstrap.cc:492] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: No bootstrap required, opened a new log
I20260620 22:56:25.974370 4015 ts_tablet_manager.cc:1397] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:25.974426 4016 ts_tablet_manager.cc:1397] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:25.974498 4015 raft_consensus.cc:348] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:25.974572 4015 raft_consensus.cc:374] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:25.974586 4015 raft_consensus.cc:729] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff35ec9148e44528f091fd65f47f25a, State: Initialized, Role: FOLLOWER
I20260620 22:56:25.974546 4016 raft_consensus.cc:348] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:25.974620 4016 raft_consensus.cc:374] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:25.974646 4016 raft_consensus.cc:729] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e7fc63ce75476d9a8000c09e1ff47f, State: Initialized, Role: FOLLOWER
I20260620 22:56:25.974633 4015 consensus_queue.cc:260] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:25.974689 4016 consensus_queue.cc:260] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:25.974716 4015 ts_tablet_manager.cc:1428] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:25.974776 4016 ts_tablet_manager.cc:1428] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:25.975131 4014 tablet_bootstrap.cc:654] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:25.975775 4014 tablet_bootstrap.cc:492] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: No bootstrap required, opened a new log
I20260620 22:56:25.975819 4014 ts_tablet_manager.cc:1397] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:56:25.975921 4014 raft_consensus.cc:348] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:25.975992 4014 raft_consensus.cc:374] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:25.976006 4014 raft_consensus.cc:729] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec97f2a36274b07acaca2c44ee4cde6, State: Initialized, Role: FOLLOWER
I20260620 22:56:25.976042 4014 consensus_queue.cc:260] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:25.976194 4014 ts_tablet_manager.cc:1428] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:26.116837 3994 raft_consensus.cc:482] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:26.116950 3994 raft_consensus.cc:504] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.117087 3994 leader_election.cc:288] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549), 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:26.117350 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14eda3fe0a764801bde66df2d167fa21" candidate_uuid: "2ff35ec9148e44528f091fd65f47f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" is_pre_election: true
I20260620 22:56:26.117362 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14eda3fe0a764801bde66df2d167fa21" candidate_uuid: "2ff35ec9148e44528f091fd65f47f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" is_pre_election: true
I20260620 22:56:26.117444 3569 raft_consensus.cc:2454] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ff35ec9148e44528f091fd65f47f25a in term 0.
I20260620 22:56:26.117445 3849 raft_consensus.cc:2454] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ff35ec9148e44528f091fd65f47f25a in term 0.
I20260620 22:56:26.117599 3643 leader_election.cc:302] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e7fc63ce75476d9a8000c09e1ff47f, 2ff35ec9148e44528f091fd65f47f25a; no voters:
I20260620 22:56:26.117707 3994 raft_consensus.cc:2781] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:26.117743 3994 raft_consensus.cc:482] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:26.117771 3994 raft_consensus.cc:3037] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:26.118604 3994 raft_consensus.cc:504] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.118722 3994 leader_election.cc:288] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [CANDIDATE]: Term 1 election: Requested vote from peers 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549), 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:26.118893 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14eda3fe0a764801bde66df2d167fa21" candidate_uuid: "2ff35ec9148e44528f091fd65f47f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec97f2a36274b07acaca2c44ee4cde6"
I20260620 22:56:26.118882 3569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14eda3fe0a764801bde66df2d167fa21" candidate_uuid: "2ff35ec9148e44528f091fd65f47f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e7fc63ce75476d9a8000c09e1ff47f"
I20260620 22:56:26.118968 3569 raft_consensus.cc:3037] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:26.118968 3849 raft_consensus.cc:3037] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:26.119717 3849 raft_consensus.cc:2454] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ff35ec9148e44528f091fd65f47f25a in term 1.
I20260620 22:56:26.119717 3569 raft_consensus.cc:2454] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ff35ec9148e44528f091fd65f47f25a in term 1.
I20260620 22:56:26.119907 3641 leader_election.cc:302] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff35ec9148e44528f091fd65f47f25a, 9ec97f2a36274b07acaca2c44ee4cde6; no voters:
I20260620 22:56:26.120024 3994 raft_consensus.cc:2781] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:26.120092 3994 raft_consensus.cc:686] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 1 LEADER]: Becoming Leader. State: Replica: 2ff35ec9148e44528f091fd65f47f25a, State: Running, Role: LEADER
I20260620 22:56:26.120146 3994 consensus_queue.cc:237] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.120712 3412 catalog_manager.cc:5495] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2). New cstate: current_term: 1 leader_uuid: "2ff35ec9148e44528f091fd65f47f25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20260620 22:56:26.163278 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:26.160526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.1.166.2:42367)
0620 22:56:26.160684 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:26.160686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:26.160695 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:26.160722 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:26.160723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:26.160735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:26.160798 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:26.160909 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:26.161430 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:26.162429 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:26.162726 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:26.162796 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:26.163003 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:26.163049 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:26.163096 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:26.163152 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 22:56:26.163178 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260620 22:56:26.166227 3569 raft_consensus.cc:1261] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Refusing update from remote peer 2ff35ec9148e44528f091fd65f47f25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:26.166227 3849 raft_consensus.cc:1261] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Refusing update from remote peer 2ff35ec9148e44528f091fd65f47f25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:26.166469 3998 consensus_queue.cc:1035] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:26.166596 3994 consensus_queue.cc:1035] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:26.178122 3410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:26.178233 3410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:26.178978 3410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: Sending DeleteTablet for 3 replicas of tablet 14eda3fe0a764801bde66df2d167fa21
I20260620 22:56:26.179330 3829 tablet_service.cc:1514] Processing DeleteTablet for tablet 14eda3fe0a764801bde66df2d167fa21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:26 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:53684
I20260620 22:56:26.179486 4004 tablet_replica.cc:330] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: stopping tablet replica
I20260620 22:56:26.179550 4004 raft_consensus.cc:2229] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:26.179595 4004 raft_consensus.cc:2258] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:26.179567 3410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:26.179698 3410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:26.179800 3549 tablet_service.cc:1514] Processing DeleteTablet for tablet 14eda3fe0a764801bde66df2d167fa21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:26 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38622
I20260620 22:56:26.179912 4003 tablet_replica.cc:330] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: stopping tablet replica
I20260620 22:56:26.179984 4003 raft_consensus.cc:2229] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:26.180042 4003 raft_consensus.cc:2258] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:26.180130 4004 ts_tablet_manager.cc:1905] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:26.180331 4003 ts_tablet_manager.cc:1905] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:26.181556 4004 ts_tablet_manager.cc:1918] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:26.181618 4004 log.cc:1196] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/14eda3fe0a764801bde66df2d167fa21
I20260620 22:56:26.181682 4003 ts_tablet_manager.cc:1918] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:26.181746 4003 log.cc:1196] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/14eda3fe0a764801bde66df2d167fa21
I20260620 22:56:26.181759 4004 ts_tablet_manager.cc:1939] T 14eda3fe0a764801bde66df2d167fa21 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting consensus metadata
I20260620 22:56:26.181934 4003 ts_tablet_manager.cc:1939] T 14eda3fe0a764801bde66df2d167fa21 P 13e7fc63ce75476d9a8000c09e1ff47f: Deleting consensus metadata
I20260620 22:56:26.182251 3399 catalog_manager.cc:4841] TS 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549): tablet 14eda3fe0a764801bde66df2d167fa21 (table test-table [id=497aa9282b91474dbc53c145be6fa32a]) successfully deleted
I20260620 22:56:26.182269 3397 catalog_manager.cc:4841] TS 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541): tablet 14eda3fe0a764801bde66df2d167fa21 (table test-table [id=497aa9282b91474dbc53c145be6fa32a]) successfully deleted
I20260620 22:56:26.182528 3689 tablet_service.cc:1514] Processing DeleteTablet for tablet 14eda3fe0a764801bde66df2d167fa21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:26 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:57906
I20260620 22:56:26.182665 4005 tablet_replica.cc:330] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: stopping tablet replica
I20260620 22:56:26.182745 4005 raft_consensus.cc:2229] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:56:26.182821 4005 raft_consensus.cc:2258] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:26.183128 4005 ts_tablet_manager.cc:1905] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:26.183135 3689 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0 (DEFAULT_TABLE table=test-table [id=1794e361d00243049b7d48defc07608c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:26.183279 3689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:26.184238 4015 tablet_bootstrap.cc:492] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: Bootstrap starting.
I20260620 22:56:26.184492 4005 ts_tablet_manager.cc:1918] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:56:26.184551 4005 log.cc:1196] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/14eda3fe0a764801bde66df2d167fa21
I20260620 22:56:26.184574 4015 tablet_bootstrap.cc:654] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:26.184759 4005 ts_tablet_manager.cc:1939] T 14eda3fe0a764801bde66df2d167fa21 P 2ff35ec9148e44528f091fd65f47f25a: Deleting consensus metadata
I20260620 22:56:26.185045 3399 catalog_manager.cc:4841] TS 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367): tablet 14eda3fe0a764801bde66df2d167fa21 (table test-table [id=497aa9282b91474dbc53c145be6fa32a]) successfully deleted
I20260620 22:56:26.185148 4015 tablet_bootstrap.cc:492] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: No bootstrap required, opened a new log
I20260620 22:56:26.185189 4015 ts_tablet_manager.cc:1397] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:26.185210 3549 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0 (DEFAULT_TABLE table=test-table [id=1794e361d00243049b7d48defc07608c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:26.185211 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0 (DEFAULT_TABLE table=test-table [id=1794e361d00243049b7d48defc07608c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:26.185312 3549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:26.185322 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:26.185294 4015 raft_consensus.cc:348] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.185461 4015 raft_consensus.cc:374] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:26.185495 4015 raft_consensus.cc:729] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff35ec9148e44528f091fd65f47f25a, State: Initialized, Role: FOLLOWER
I20260620 22:56:26.185547 4015 consensus_queue.cc:260] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.185642 4015 ts_tablet_manager.cc:1428] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:26.186627 4016 tablet_bootstrap.cc:492] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f: Bootstrap starting.
I20260620 22:56:26.186836 4014 tablet_bootstrap.cc:492] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: Bootstrap starting.
I20260620 22:56:26.187048 4016 tablet_bootstrap.cc:654] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:26.187155 4014 tablet_bootstrap.cc:654] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:26.187610 4016 tablet_bootstrap.cc:492] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f: No bootstrap required, opened a new log
I20260620 22:56:26.187665 4016 ts_tablet_manager.cc:1397] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:26.187757 4014 tablet_bootstrap.cc:492] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: No bootstrap required, opened a new log
I20260620 22:56:26.187805 4014 ts_tablet_manager.cc:1397] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:26.187788 4016 raft_consensus.cc:348] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.187865 4016 raft_consensus.cc:374] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:26.187891 4016 raft_consensus.cc:729] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e7fc63ce75476d9a8000c09e1ff47f, State: Initialized, Role: FOLLOWER
I20260620 22:56:26.187938 4016 consensus_queue.cc:260] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.187923 4014 raft_consensus.cc:348] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.187987 4014 raft_consensus.cc:374] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:26.188006 4014 raft_consensus.cc:729] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec97f2a36274b07acaca2c44ee4cde6, State: Initialized, Role: FOLLOWER
I20260620 22:56:26.188035 4016 ts_tablet_manager.cc:1428] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:26.188055 4014 consensus_queue.cc:260] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.188144 4014 ts_tablet_manager.cc:1428] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:26.217269 3926 raft_consensus.cc:482] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:26.217386 3926 raft_consensus.cc:504] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.217558 3926 leader_election.cc:288] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367), 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:26.217793 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc1a5d7e7c4dadb1138ceed17645e0" candidate_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" is_pre_election: true
I20260620 22:56:26.217784 3709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc1a5d7e7c4dadb1138ceed17645e0" candidate_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff35ec9148e44528f091fd65f47f25a" is_pre_election: true
I20260620 22:56:26.217880 3849 raft_consensus.cc:2454] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e7fc63ce75476d9a8000c09e1ff47f in term 0.
I20260620 22:56:26.217882 3709 raft_consensus.cc:2454] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e7fc63ce75476d9a8000c09e1ff47f in term 0.
I20260620 22:56:26.218043 3501 leader_election.cc:302] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e7fc63ce75476d9a8000c09e1ff47f, 9ec97f2a36274b07acaca2c44ee4cde6; no voters:
I20260620 22:56:26.218135 3926 raft_consensus.cc:2781] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:26.218173 3926 raft_consensus.cc:482] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:26.218189 3926 raft_consensus.cc:3037] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:26.218943 3926 raft_consensus.cc:504] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.219076 3926 leader_election.cc:288] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [CANDIDATE]: Term 1 election: Requested vote from peers 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367), 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541)
I20260620 22:56:26.219220 3709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc1a5d7e7c4dadb1138ceed17645e0" candidate_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ff35ec9148e44528f091fd65f47f25a"
I20260620 22:56:26.219292 3709 raft_consensus.cc:3037] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:26.219856 3709 raft_consensus.cc:2454] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e7fc63ce75476d9a8000c09e1ff47f in term 1.
I20260620 22:56:26.219964 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcc1a5d7e7c4dadb1138ceed17645e0" candidate_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec97f2a36274b07acaca2c44ee4cde6"
I20260620 22:56:26.220045 3849 raft_consensus.cc:3037] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:26.220060 3503 leader_election.cc:302] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e7fc63ce75476d9a8000c09e1ff47f, 2ff35ec9148e44528f091fd65f47f25a; no voters:
I20260620 22:56:26.220156 3926 raft_consensus.cc:2781] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:26.220211 3926 raft_consensus.cc:686] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [term 1 LEADER]: Becoming Leader. State: Replica: 13e7fc63ce75476d9a8000c09e1ff47f, State: Running, Role: LEADER
I20260620 22:56:26.220269 3926 consensus_queue.cc:237] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } }
I20260620 22:56:26.220789 3849 raft_consensus.cc:2454] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e7fc63ce75476d9a8000c09e1ff47f in term 1.
I20260620 22:56:26.220944 3410 catalog_manager.cc:5495] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1). New cstate: current_term: 1 leader_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e7fc63ce75476d9a8000c09e1ff47f" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 } health_report { overall_health: UNKNOWN } } }
I20260620 22:56:26.229813 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:26.227030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.1.166.1:44549)
0620 22:56:26.227180 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:26.227182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:26.227190 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:26.227211 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:26.227211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:26.227221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:26.227284 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:26.227562 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:26.228082 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:26.228976 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:26.229228 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:26.229300 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:26.229510 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:26.229557 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:26.229613 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:26.229677 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 22:56:26.229705 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260620 22:56:26.234965 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:26.232952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:47493 (local address 127.1.166.1:44549)
0620 22:56:26.233006 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:26.233008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:26.233016 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:26.233047 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:26.233047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:26.233058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:26.233097 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:26.233229 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:26.233700 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:26.234392 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:26.234565 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:26.234617 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:26.234727 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:26.234760 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:26.234799 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:26.234849 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 22:56:26.234870 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":37}
I20260620 22:56:26.234965 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:26.232781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44549 (local address 127.1.166.1:47493)
0620 22:56:26.232935 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:26.232946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:26.232998 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:26.233102 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:26.233103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:26.233104 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:56:26.233171 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:26.233180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:26.233761 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:26.233762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:26.234303 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:26.234304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:26.234622 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:26.234623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:26.234671 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:26.234675 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:56:26.234806 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:56:26.234808 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:56:26.234849 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 22:56:26.234889 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260620 22:56:26.236865 3709 raft_consensus.cc:1261] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Refusing update from remote peer 13e7fc63ce75476d9a8000c09e1ff47f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:26.236861 3849 raft_consensus.cc:1261] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Refusing update from remote peer 13e7fc63ce75476d9a8000c09e1ff47f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:26.237077 3984 consensus_queue.cc:1035] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ff35ec9148e44528f091fd65f47f25a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:26.237191 3926 consensus_queue.cc:1035] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 13e7fc63ce75476d9a8000c09e1ff47f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ec97f2a36274b07acaca2c44ee4cde6" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:56:26.241834 3982 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 22:56:27.764490 3411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:27.764601 3411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:27.765482 3411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 187a8cbeed6941aeac53e329412146d8: Sending DeleteTablet for 3 replicas of tablet 3fcc1a5d7e7c4dadb1138ceed17645e0
I20260620 22:56:27.765720 3549 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:27 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38622
I20260620 22:56:27.765743 3829 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:27 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:53684
I20260620 22:56:27.765760 3689 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:27 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:57906
I20260620 22:56:27.765877 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 3484
I20260620 22:56:27.765942 4059 tablet_replica.cc:330] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: stopping tablet replica
I20260620 22:56:27.767081 4059 raft_consensus.cc:2229] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:27.767127 4059 raft_consensus.cc:2258] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:27.767398 4059 ts_tablet_manager.cc:1905] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:27.768358 4058 tablet_replica.cc:330] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: stopping tablet replica
I20260620 22:56:27.768430 4058 raft_consensus.cc:2229] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:27.768471 4058 raft_consensus.cc:2258] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:27.768734 4058 ts_tablet_manager.cc:1905] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:27.769066 4059 ts_tablet_manager.cc:1918] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:56:27.769127 4059 log.cc:1196] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/3fcc1a5d7e7c4dadb1138ceed17645e0
I20260620 22:56:27.769331 4059 ts_tablet_manager.cc:1939] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 9ec97f2a36274b07acaca2c44ee4cde6: Deleting consensus metadata
I20260620 22:56:27.769678 3397 catalog_manager.cc:4841] TS 9ec97f2a36274b07acaca2c44ee4cde6 (127.1.166.3:37541): tablet 3fcc1a5d7e7c4dadb1138ceed17645e0 (table test-table [id=1794e361d00243049b7d48defc07608c]) successfully deleted
I20260620 22:56:27.770102 4058 ts_tablet_manager.cc:1918] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:56:27.770150 4058 log.cc:1196] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/3fcc1a5d7e7c4dadb1138ceed17645e0
I20260620 22:56:27.770309 4058 ts_tablet_manager.cc:1939] T 3fcc1a5d7e7c4dadb1138ceed17645e0 P 2ff35ec9148e44528f091fd65f47f25a: Deleting consensus metadata
I20260620 22:56:27.770771 3399 catalog_manager.cc:4841] TS 2ff35ec9148e44528f091fd65f47f25a (127.1.166.2:42367): tablet 3fcc1a5d7e7c4dadb1138ceed17645e0 (table test-table [id=1794e361d00243049b7d48defc07608c]) successfully deleted
W20260620 22:56:27.772944 3399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:44549 (error 108)
I20260620 22:56:27.773592 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 3624
I20260620 22:56:27.774359 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:27.773738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44549 (local address 127.0.0.1:38642)
0620 22:56:27.774250 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:27.774299 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":70,"threads_started":1}
W20260620 22:56:27.774467 3399 catalog_manager.cc:4584] TS 13e7fc63ce75476d9a8000c09e1ff47f (127.1.166.1:44549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fcc1a5d7e7c4dadb1138ceed17645e0: Network error: Client connection negotiation failed: client connection to 127.1.166.1:44549: connect: Connection refused (error 111)
I20260620 22:56:27.778976 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 3764
I20260620 22:56:27.784178 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 3380
2026-06-20T22:56:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5554 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:27 dist-test-slave-rdl9 krb5kdc[4064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:27 dist-test-slave-rdl9 krb5kdc[4064](info): set up 2 sockets
Jun 20 22:56:27 dist-test-slave-rdl9 krb5kdc[4064](info): commencing operation
krb5kdc: starting...
W20260620 22:56:28.774001 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:28 dist-test-slave-rdl9 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:28.885476 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46613
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:28.967391 4077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:28.967650 4077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:28.967686 4077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:28.969018 4077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:28.969086 4077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:28.969106 4077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:28.969123 4077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:28.969137 4077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:28.970916 4077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:46613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:28.971244 4077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:28.971469 4077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:28.977087 4085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:28.977123 4083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:28.977102 4082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:28.977466 4077 server_base.cc:1034] running on GCE node
I20260620 22:56:28.977732 4077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:28.978066 4077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:28.979240 4077 hybrid_clock.cc:648] HybridClock initialized: now 1781996188979215 us; error 61 us; skew 500 ppm
Jun 20 22:56:28 dist-test-slave-rdl9 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996188, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:28.982187 4077 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:28.982762 4077 webserver.cc:460] Webserver started at http://127.1.166.62:38691/ using document root <none> and password file <none>
I20260620 22:56:28.982996 4077 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:28.983050 4077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:28.983204 4077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:28.984447 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "58197781c5494627a67175bf844a8490"
format_stamp: "Formatted at 2026-06-20 22:56:28 on dist-test-slave-rdl9"
server_key: "5a61d4bc1867335561a685bdbe6d07e1"
server_key_iv: "6902807d6e53caa95d240eeaf6f63ec0"
server_key_version: "encryptionkey@0"
I20260620 22:56:28.984885 4077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "58197781c5494627a67175bf844a8490"
format_stamp: "Formatted at 2026-06-20 22:56:28 on dist-test-slave-rdl9"
server_key: "5a61d4bc1867335561a685bdbe6d07e1"
server_key_iv: "6902807d6e53caa95d240eeaf6f63ec0"
server_key_version: "encryptionkey@0"
I20260620 22:56:28.986546 4077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:28.987779 4092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:28.988124 4077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:28.988245 4077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "58197781c5494627a67175bf844a8490"
format_stamp: "Formatted at 2026-06-20 22:56:28 on dist-test-slave-rdl9"
server_key: "5a61d4bc1867335561a685bdbe6d07e1"
server_key_iv: "6902807d6e53caa95d240eeaf6f63ec0"
server_key_version: "encryptionkey@0"
I20260620 22:56:28.988343 4077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:29.016759 4077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:29.019166 4077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:29.019321 4077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:29.023296 4077 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:46613
I20260620 22:56:29.023433 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:46613 every 8 connection(s)
I20260620 22:56:29.023728 4077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:29.024605 4145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:29.027407 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490: Bootstrap starting.
I20260620 22:56:29.027951 4145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:29.028182 4145 log.cc:824] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:29.028884 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490: No bootstrap required, opened a new log
I20260620 22:56:29.030537 4145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58197781c5494627a67175bf844a8490" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46613 } }
I20260620 22:56:29.030709 4145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:29.030742 4145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58197781c5494627a67175bf844a8490, State: Initialized, Role: FOLLOWER
I20260620 22:56:29.030752 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 4077
I20260620 22:56:29.030851 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:29.030893 4145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58197781c5494627a67175bf844a8490" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46613 } }
I20260620 22:56:29.030977 4145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:29.031015 4145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:29.031042 1688 external_mini_cluster.cc:1428] Setting key 704bfe96324d197f4b8caf9794472dcb
I20260620 22:56:29.031064 4145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:29.032194 4145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58197781c5494627a67175bf844a8490" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46613 } }
I20260620 22:56:29.032322 4145 leader_election.cc:302] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58197781c5494627a67175bf844a8490; no voters:
I20260620 22:56:29.032483 4145 leader_election.cc:288] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:29.032539 4150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:29.032686 4150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [term 1 LEADER]: Becoming Leader. State: Replica: 58197781c5494627a67175bf844a8490, State: Running, Role: LEADER
I20260620 22:56:29.032732 4145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:29.032781 4150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58197781c5494627a67175bf844a8490" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46613 } }
I20260620 22:56:29.033138 4150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58197781c5494627a67175bf844a8490. Latest consensus state: current_term: 1 leader_uuid: "58197781c5494627a67175bf844a8490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58197781c5494627a67175bf844a8490" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46613 } } }
I20260620 22:56:29.033213 4150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:29.033998 4152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58197781c5494627a67175bf844a8490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58197781c5494627a67175bf844a8490" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46613 } } }
I20260620 22:56:29.034171 4152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490 [sys.catalog]: This master's current role is: LEADER
W20260620 22:56:29.034482 4166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:29.034535 4166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:29.034584 4157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:29.034732 4157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:29.036285 4157 catalog_manager.cc:1347] Generated new cluster ID: 0a2432350c1c41189957aa492fe93bdf
I20260620 22:56:29.036342 4157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:29.041564 4157 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:29.042192 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:29.031537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.1.166.62:46613)
0620 22:56:29.031800 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:29.031812 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:29.031852 (+ 40us) server_negotiation.cc:407] Connection header received
0620 22:56:29.031986 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:29.031993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:29.032062 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:29.032212 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:29.033054 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.034557 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.035674 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.036148 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.036249 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:29.038659 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:29.038664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:29.038664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:29.038678 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:29.040309 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:29.040680 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:29.040684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:29.040684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:29.040800 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:29.040984 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:29.040985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:29.040985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:29.041395 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:29.041590 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:29.041786 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 22:56:29.042027 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":202,"thread_start_us":145,"threads_started":1}
I20260620 22:56:29.042289 4157 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:29.048487 4157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58197781c5494627a67175bf844a8490: Generated new TSK 0
I20260620 22:56:29.048698 4157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:29.075289 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46613
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:29.168855 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:29.169099 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:29.169137 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:29.170686 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:29.170761 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:29.170809 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:29.172464 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:29.172878 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:29.173103 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:29.179018 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:29.179024 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:29.179023 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:29.179325 4171 server_base.cc:1034] running on GCE node
I20260620 22:56:29.179509 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:29.179747 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:29.180907 4171 hybrid_clock.cc:648] HybridClock initialized: now 1781996189180890 us; error 31 us; skew 500 ppm
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996189, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:29.183495 4171 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:29.184007 4171 webserver.cc:460] Webserver started at http://127.1.166.1:41591/ using document root <none> and password file <none>
I20260620 22:56:29.184229 4171 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:29.184281 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:29.184418 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:29.185539 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "9e3327b505864332bb4a4a7dfa540f77"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "77c4c1ae32c5aaf01002457032f81987"
server_key_iv: "e2a5bc8927d6c5c4d85b343daff43f37"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.185909 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "9e3327b505864332bb4a4a7dfa540f77"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "77c4c1ae32c5aaf01002457032f81987"
server_key_iv: "e2a5bc8927d6c5c4d85b343daff43f37"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.187426 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:29.188305 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.188509 4171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:56:29.188602 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "9e3327b505864332bb4a4a7dfa540f77"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "77c4c1ae32c5aaf01002457032f81987"
server_key_iv: "e2a5bc8927d6c5c4d85b343daff43f37"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.188670 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:29.217353 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:29.219739 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:29.219888 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:29.220315 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:29.220736 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:29.220777 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.220810 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:29.220834 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.226588 4171 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:37295
I20260620 22:56:29.226641 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:37295 every 8 connection(s)
I20260620 22:56:29.226989 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:56:29.232324 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 4171
I20260620 22:56:29.232414 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:29.232601 1688 external_mini_cluster.cc:1428] Setting key 5deeeb8418ef80da3a286f5a18d233ad
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996189, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:29.237183 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:29.227809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:35651 (local address 127.1.166.62:46613)
0620 22:56:29.227879 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:29.227881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:29.228171 (+ 290us) server_negotiation.cc:407] Connection header received
0620 22:56:29.228675 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:29.228677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:29.228690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:29.228756 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:29.229755 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.230271 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.231350 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.231558 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.231621 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:29.234257 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:29.234259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:29.234259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:29.234272 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:29.235630 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:29.236105 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:29.236107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:29.236107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:29.236167 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:29.236440 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:29.236441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:29.236441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:29.236567 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:29.236621 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:29.237034 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 22:56:29.237114 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:56:29.237483 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:29.227720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46613 (local address 127.1.166.1:35651)
0620 22:56:29.227990 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:29.228033 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:29.228530 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:29.228825 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:29.228829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:29.229431 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:29.229641 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:29.229645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.230419 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.230420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:29.231252 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:29.231254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.231629 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.231630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:29.231735 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:29.232257 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:29.232265 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:29.234151 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:29.235715 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:29.235724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:29.235727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:29.236018 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:29.236228 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:29.236228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:29.236229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:29.236369 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:29.236658 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:29.236667 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:29.236857 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 22:56:29.237007 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 22:56:29.237227 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":185,"threads_started":1}
I20260620 22:56:29.238027 4300 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46613
I20260620 22:56:29.238135 4300 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:29.238324 4300 heartbeater.cc:511] Master 127.1.166.62:46613 requested a full tablet report, sending...
I20260620 22:56:29.239061 4109 ts_manager.cc:194] Registered new tserver with Master: 9e3327b505864332bb4a4a7dfa540f77 (127.1.166.1:37295)
I20260620 22:56:29.239981 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:35651
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:29.257109 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46613
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:29.341665 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:29.341917 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:29.341956 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:29.343349 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:29.343415 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:29.343458 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:29.345182 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:29.345511 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:29.345762 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:29.350903 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:29.350888 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:29.350906 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:29.351217 4305 server_base.cc:1034] running on GCE node
I20260620 22:56:29.351419 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:29.351680 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:29.352825 4305 hybrid_clock.cc:648] HybridClock initialized: now 1781996189352801 us; error 35 us; skew 500 ppm
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996189, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:29.355638 4305 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:29.356129 4305 webserver.cc:460] Webserver started at http://127.1.166.2:35829/ using document root <none> and password file <none>
I20260620 22:56:29.356380 4305 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:29.356433 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:29.356557 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:29.357630 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "f09ca082c077422dbaef5ff5ad43a16f"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "e7af87bacf866481e6786aed67d9837b"
server_key_iv: "b628b784b34d8ad2f9447e87b56d27d2"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.357954 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "f09ca082c077422dbaef5ff5ad43a16f"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "e7af87bacf866481e6786aed67d9837b"
server_key_iv: "b628b784b34d8ad2f9447e87b56d27d2"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.359301 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:56:29.360103 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.360304 4305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:29.360402 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "f09ca082c077422dbaef5ff5ad43a16f"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "e7af87bacf866481e6786aed67d9837b"
server_key_iv: "b628b784b34d8ad2f9447e87b56d27d2"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.360467 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:29.389362 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:29.391290 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:29.391436 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:29.391672 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:29.391992 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:29.392026 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.392055 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:29.392078 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.397751 4305 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:41017
I20260620 22:56:29.397801 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:41017 every 8 connection(s)
I20260620 22:56:29.398113 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:29.402609 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 4305
I20260620 22:56:29.402695 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:29.402861 1688 external_mini_cluster.cc:1428] Setting key cd85ad90e5ac4eabcc5240c74df3a951
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996189, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:29.407469 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:29.398871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:45275 (local address 127.1.166.62:46613)
0620 22:56:29.398941 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:29.398943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:29.399123 (+ 180us) server_negotiation.cc:407] Connection header received
0620 22:56:29.399611 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:29.399612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:29.399623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:29.399692 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:29.400576 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.401069 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.402010 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.402188 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.402231 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:29.404744 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:29.404746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:29.404747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:29.404754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:29.406076 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:29.406548 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:29.406549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:29.406549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:29.406602 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:29.406825 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:29.406826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:29.406827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:29.406949 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:29.407007 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:29.407282 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 22:56:29.407372 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:56:29.407677 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:29.398840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46613 (local address 127.1.166.2:45275)
0620 22:56:29.398995 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:29.399037 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:29.399505 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:29.399748 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:29.399751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:29.400306 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:29.400497 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:29.400502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.401145 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.401145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:29.401949 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:29.401951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.402241 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.402242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:29.402366 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:29.402926 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:29.402937 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:29.404638 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:29.406162 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:29.406172 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:29.406174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:29.406473 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:29.406655 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:29.406655 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:29.406656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:29.406767 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:29.407006 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:29.407013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:29.407163 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 22:56:29.407256 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 22:56:29.407441 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260620 22:56:29.408449 4434 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46613
I20260620 22:56:29.408550 4434 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:29.408730 4434 heartbeater.cc:511] Master 127.1.166.62:46613 requested a full tablet report, sending...
I20260620 22:56:29.409204 4109 ts_manager.cc:194] Registered new tserver with Master: f09ca082c077422dbaef5ff5ad43a16f (127.1.166.2:41017)
I20260620 22:56:29.409586 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:45275
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:29.428483 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46613
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:29.516287 4439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:29.516574 4439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:29.516607 4439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:29.518379 4439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:29.518486 4439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:29.518553 4439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:29.520951 4439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:29.521250 4439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:29.521495 4439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:29.526692 4447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:29.526660 4445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:29.526664 4444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:29.526703 4439 server_base.cc:1034] running on GCE node
I20260620 22:56:29.527140 4439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:29.527386 4439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:29.528577 4439 hybrid_clock.cc:648] HybridClock initialized: now 1781996189528558 us; error 35 us; skew 500 ppm
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996189, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:29.531898 4439 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:29.532418 4439 webserver.cc:460] Webserver started at http://127.1.166.3:41327/ using document root <none> and password file <none>
I20260620 22:56:29.532642 4439 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:29.532692 4439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:29.532826 4439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:29.534106 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "d02b4df9b77d4028a8b92ffd93f73a94"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "db30ab26ea6ccee7bea34ddf1ab75f70"
server_key_iv: "9232ef4026bd629650dcaaf6b0126c77"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.534512 4439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "d02b4df9b77d4028a8b92ffd93f73a94"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "db30ab26ea6ccee7bea34ddf1ab75f70"
server_key_iv: "9232ef4026bd629650dcaaf6b0126c77"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.535892 4439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:29.536721 4454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.536954 4439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:29.537182 4439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "d02b4df9b77d4028a8b92ffd93f73a94"
format_stamp: "Formatted at 2026-06-20 22:56:29 on dist-test-slave-rdl9"
server_key: "db30ab26ea6ccee7bea34ddf1ab75f70"
server_key_iv: "9232ef4026bd629650dcaaf6b0126c77"
server_key_version: "encryptionkey@0"
I20260620 22:56:29.537356 4439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:29.570433 4439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:29.572541 4439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:29.572688 4439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:29.572933 4439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:29.573251 4439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:29.573285 4439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.573313 4439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:29.573330 4439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:29.579037 4439 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:44891
I20260620 22:56:29.579083 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:44891 every 8 connection(s)
I20260620 22:56:29.579393 4439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:29.584692 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 4439
I20260620 22:56:29.584820 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:29.585021 1688 external_mini_cluster.cc:1428] Setting key f11a810cc046e4cd948967f5309d755a
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996189, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:29.590380 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:29.580107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:51103 (local address 127.1.166.62:46613)
0620 22:56:29.580167 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:29.580169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:29.580356 (+ 187us) server_negotiation.cc:407] Connection header received
0620 22:56:29.580882 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:29.580883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:29.580896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:29.580950 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:29.581928 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.582469 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.583442 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.583642 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.583699 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:29.586491 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:29.586493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:29.586493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:29.586501 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:29.588275 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:29.589004 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:29.589006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:29.589006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:29.589078 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:29.589422 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:29.589423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:29.589424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:29.589618 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:29.589729 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:29.590142 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 22:56:29.590252 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:56:29.590596 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:29.580071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46613 (local address 127.1.166.3:51103)
0620 22:56:29.580245 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:29.580281 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:29.580774 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:29.581029 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:29.581033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:29.581627 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:29.581825 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:29.581830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.582549 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.582550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:29.583351 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:29.583354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:29.583709 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:29.583710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:29.583838 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:29.584363 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:29.584379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:29.586393 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:29.588413 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:29.588426 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:29.588429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:29.588904 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:29.589141 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:29.589142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:29.589142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:29.589307 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:29.589733 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:29.589741 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:29.589961 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 22:56:29.590086 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 22:56:29.590292 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260620 22:56:29.591401 4568 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46613
I20260620 22:56:29.591516 4568 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:29.591709 4568 heartbeater.cc:511] Master 127.1.166.62:46613 requested a full tablet report, sending...
I20260620 22:56:29.592204 4109 ts_manager.cc:194] Registered new tserver with Master: d02b4df9b77d4028a8b92ffd93f73a94 (127.1.166.3:44891)
I20260620 22:56:29.592634 4109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:51103
I20260620 22:56:29.597263 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:56:29.607475 4149 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:29.606693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.1.166.62:46613)
0620 22:56:29.606747 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:29.606749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:29.606795 (+ 46us) server_negotiation.cc:407] Connection header received
0620 22:56:29.606867 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:29.606868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:29.606878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:29.606920 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:29.607401 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50866: BlockingRecv error: recv got EOF from 127.0.0.1:50866 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:29.608423 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 4171
I20260620 22:56:29.612632 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 4305
I20260620 22:56:29.616474 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 4439
I20260620 22:56:29.620586 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 4077
2026-06-20T22:56:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1834 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4580](info): set up 2 sockets
Jun 20 22:56:29 dist-test-slave-rdl9 krb5kdc[4580](info): commencing operation
krb5kdc: starting...
W20260620 22:56:30.590258 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:30 dist-test-slave-rdl9 krb5kdc[4580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:30.707672 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35497
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:30.794978 4593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:30.795209 4593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:30.795243 4593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:30.796518 4593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:30.796567 4593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:30.796586 4593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:30.796607 4593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:30.796622 4593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:30.798251 4593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:35497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:30.798614 4593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:30.798867 4593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:30.803666 4601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:30.803661 4598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:30.803671 4599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:30.803930 4593 server_base.cc:1034] running on GCE node
I20260620 22:56:30.804165 4593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:30.804458 4593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:30.805645 4593 hybrid_clock.cc:648] HybridClock initialized: now 1781996190805625 us; error 35 us; skew 500 ppm
Jun 20 22:56:30 dist-test-slave-rdl9 krb5kdc[4580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996190, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:30.808723 4593 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:30.809263 4593 webserver.cc:460] Webserver started at http://127.1.166.62:35253/ using document root <none> and password file <none>
I20260620 22:56:30.809505 4593 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:30.809556 4593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:30.809688 4593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:30.810891 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2"
format_stamp: "Formatted at 2026-06-20 22:56:30 on dist-test-slave-rdl9"
server_key: "80317395c9d3c6a6a6f2f7d601e664a3"
server_key_iv: "f3769c4f577578dfc422521c1abbf1ed"
server_key_version: "encryptionkey@0"
I20260620 22:56:30.811235 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2"
format_stamp: "Formatted at 2026-06-20 22:56:30 on dist-test-slave-rdl9"
server_key: "80317395c9d3c6a6a6f2f7d601e664a3"
server_key_iv: "f3769c4f577578dfc422521c1abbf1ed"
server_key_version: "encryptionkey@0"
I20260620 22:56:30.812640 4593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:30.813524 4608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:30.813862 4593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:30.813958 4593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2"
format_stamp: "Formatted at 2026-06-20 22:56:30 on dist-test-slave-rdl9"
server_key: "80317395c9d3c6a6a6f2f7d601e664a3"
server_key_iv: "f3769c4f577578dfc422521c1abbf1ed"
server_key_version: "encryptionkey@0"
I20260620 22:56:30.814039 4593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:30.829758 4593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:30.832047 4593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:30.832202 4593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:30.836020 4593 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:35497
I20260620 22:56:30.836054 4660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:35497 every 8 connection(s)
I20260620 22:56:30.836520 4593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:30.837379 4661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:30.839716 4661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2: Bootstrap starting.
I20260620 22:56:30.840291 4661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:30.840526 4661 log.cc:824] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:30.841313 4661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2: No bootstrap required, opened a new log
I20260620 22:56:30.842967 4661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35497 } }
I20260620 22:56:30.843161 4661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:30.843199 4661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b34bb5496574a77a8cc5fdf2c67d0b2, State: Initialized, Role: FOLLOWER
I20260620 22:56:30.843286 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 4593
I20260620 22:56:30.843372 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:30.843359 4661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35497 } }
I20260620 22:56:30.843441 4661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:30.843480 4661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:30.843528 4661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:30.843540 1688 external_mini_cluster.cc:1428] Setting key aa1b59bfe3f9ec8c8cd8ddfc2bcc4e89
I20260620 22:56:30.844904 4661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35497 } }
I20260620 22:56:30.845041 4661 leader_election.cc:302] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b34bb5496574a77a8cc5fdf2c67d0b2; no voters:
I20260620 22:56:30.845207 4661 leader_election.cc:288] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:30.845252 4666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:30.845391 4666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [term 1 LEADER]: Becoming Leader. State: Replica: 1b34bb5496574a77a8cc5fdf2c67d0b2, State: Running, Role: LEADER
I20260620 22:56:30.845485 4661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:30.845510 4666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35497 } }
I20260620 22:56:30.845862 4667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35497 } } }
I20260620 22:56:30.845957 4667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:30.845860 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b34bb5496574a77a8cc5fdf2c67d0b2. Latest consensus state: current_term: 1 leader_uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34bb5496574a77a8cc5fdf2c67d0b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35497 } } }
I20260620 22:56:30.846056 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:30.846158 4672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:30.846720 4672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:30.848263 4672 catalog_manager.cc:1347] Generated new cluster ID: 87ec6dcb595b454da63cc4c51da6efeb
I20260620 22:56:30.848320 4672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:56:30 dist-test-slave-rdl9 krb5kdc[4580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:30.855171 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:30.844553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.1.166.62:35497)
0620 22:56:30.844759 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:30.844770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:30.844819 (+ 49us) server_negotiation.cc:407] Connection header received
0620 22:56:30.844950 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:30.844957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:30.845020 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:30.845178 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:30.846128 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:30.847488 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:30.848347 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:30.848760 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:30.848854 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:30.851160 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:30.851164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:30.851164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:30.851175 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:30.853255 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:30.853863 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:30.853866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:30.853866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:30.854013 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:30.854224 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:30.854225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:30.854226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:30.854535 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:30.854712 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:30.854811 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 22:56:30.854967 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260620 22:56:30.856098 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:30.858033 4672 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:30.858665 4672 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:30.866376 4672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b34bb5496574a77a8cc5fdf2c67d0b2: Generated new TSK 0
I20260620 22:56:30.866564 4672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:30.876569 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:30.873211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.1.166.62:35497)
0620 22:56:30.873263 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:30.873264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:30.873401 (+ 137us) server_negotiation.cc:407] Connection header received
0620 22:56:30.873472 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:30.873473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:30.873485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:30.873532 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:30.873896 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:30.874460 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:30.875195 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:30.875417 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:30.875483 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:30.876120 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:30.876121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:30.876122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:30.876329 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:30.876346 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:30.876352 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:30.876396 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:30.876458 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 22:56:30.876485 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:56:30.879729 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 4593
2026-06-20T22:56:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1256 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:30 dist-test-slave-rdl9 krb5kdc[4694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:30 dist-test-slave-rdl9 krb5kdc[4694](info): set up 2 sockets
Jun 20 22:56:30 dist-test-slave-rdl9 krb5kdc[4694](info): commencing operation
krb5kdc: starting...
W20260620 22:56:31.862004 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:31 dist-test-slave-rdl9 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:31.982019 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38035
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:38035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:32.070457 4707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:32.070705 4707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:32.070749 4707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:32.072103 4707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:32.072171 4707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:32.072189 4707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:32.072206 4707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:32.072222 4707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:32.074013 4707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:38035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:32.074329 4707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:32.074595 4707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:56:32.079540 4707 server_base.cc:1034] running on GCE node
W20260620 22:56:32.079542 4712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:32.079581 4713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:32.079684 4715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:32.079914 4707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:32.080220 4707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:32.081354 4707 hybrid_clock.cc:648] HybridClock initialized: now 1781996192081343 us; error 39 us; skew 500 ppm
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:32.086716 4707 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:32.087164 4707 webserver.cc:460] Webserver started at http://127.1.166.62:33257/ using document root <none> and password file <none>
I20260620 22:56:32.087378 4707 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:32.087425 4707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:32.087545 4707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:32.088552 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "3d473b658f714e458dc3cd148930b797"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "56b8d644e160a573d316b38f43ea0e27"
server_key_iv: "c4d87d9d0b5f0804bc39f322056eee28"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.088883 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "3d473b658f714e458dc3cd148930b797"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "56b8d644e160a573d316b38f43ea0e27"
server_key_iv: "c4d87d9d0b5f0804bc39f322056eee28"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.090318 4707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:56:32.091369 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.091679 4707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:32.091799 4707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "3d473b658f714e458dc3cd148930b797"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "56b8d644e160a573d316b38f43ea0e27"
server_key_iv: "c4d87d9d0b5f0804bc39f322056eee28"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.091874 4707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:32.113401 4707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:32.115592 4707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:32.115736 4707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:32.119822 4707 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:38035
I20260620 22:56:32.119865 4774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:38035 every 8 connection(s)
I20260620 22:56:32.120206 4707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:32.121178 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:32.123541 4775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797: Bootstrap starting.
I20260620 22:56:32.124120 4775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:32.124372 4775 log.cc:824] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:32.125214 4775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797: No bootstrap required, opened a new log
I20260620 22:56:32.126925 4775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d473b658f714e458dc3cd148930b797" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38035 } }
I20260620 22:56:32.127099 4775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:32.127121 4775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d473b658f714e458dc3cd148930b797, State: Initialized, Role: FOLLOWER
I20260620 22:56:32.127266 4775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d473b658f714e458dc3cd148930b797" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38035 } }
I20260620 22:56:32.127336 4775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:32.127357 4775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:32.127389 4775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:32.127641 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 4707
I20260620 22:56:32.127734 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:32.127892 1688 external_mini_cluster.cc:1428] Setting key 7c92fc6ecb4a8f59f93c99a569c0240d
I20260620 22:56:32.128937 4775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d473b658f714e458dc3cd148930b797" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38035 } }
I20260620 22:56:32.129062 4775 leader_election.cc:302] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d473b658f714e458dc3cd148930b797; no voters:
I20260620 22:56:32.129240 4775 leader_election.cc:288] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:32.129329 4780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:32.129513 4780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [term 1 LEADER]: Becoming Leader. State: Replica: 3d473b658f714e458dc3cd148930b797, State: Running, Role: LEADER
I20260620 22:56:32.129519 4775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:32.129653 4780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d473b658f714e458dc3cd148930b797" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38035 } }
I20260620 22:56:32.130779 4781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d473b658f714e458dc3cd148930b797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d473b658f714e458dc3cd148930b797" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38035 } } }
I20260620 22:56:32.130896 4781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:32.131301 4796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:32.131341 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d473b658f714e458dc3cd148930b797. Latest consensus state: current_term: 1 leader_uuid: "3d473b658f714e458dc3cd148930b797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d473b658f714e458dc3cd148930b797" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38035 } } }
I20260620 22:56:32.131492 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:32.131932 4796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:32.133801 4796 catalog_manager.cc:1347] Generated new cluster ID: 8005168193384ec38bc0801a868740db
I20260620 22:56:32.133873 4796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:32.141264 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:32.128501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.1.166.62:38035)
0620 22:56:32.128747 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:32.128758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:32.128800 (+ 42us) server_negotiation.cc:407] Connection header received
0620 22:56:32.128939 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:32.128948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:32.129014 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:32.129172 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:32.130064 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.131835 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.132983 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.133552 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.133672 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:32.137172 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:32.137179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:32.137180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:32.137197 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:32.139296 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.139738 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.139741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.139741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.139887 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.140078 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.140078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.140079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.140445 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:32.140617 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:32.140696 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 22:56:32.140984 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":135,"threads_started":1}
I20260620 22:56:32.153549 4796 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:32.154317 4796 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:32.162014 4796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d473b658f714e458dc3cd148930b797: Generated new TSK 0
I20260620 22:56:32.162245 4796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:32.195585 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38035
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:32.284816 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:32.285075 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:32.285110 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:32.286535 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:32.286604 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:32.286648 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:32.288386 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:32.288687 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:32.288928 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:32.294219 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:32.294152 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:32.294145 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:32.294456 4801 server_base.cc:1034] running on GCE node
I20260620 22:56:32.294739 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:32.294993 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:32.296140 4801 hybrid_clock.cc:648] HybridClock initialized: now 1781996192296130 us; error 28 us; skew 500 ppm
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:32.298941 4801 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:32.299400 4801 webserver.cc:460] Webserver started at http://127.1.166.1:32961/ using document root <none> and password file <none>
I20260620 22:56:32.299587 4801 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:32.299628 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:32.299717 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:32.300766 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "e1329cfe5acd40808df49d3504fcfef9"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "19deed9f94c893743db435b1394c2914"
server_key_iv: "c3345eda368a7652e64c58fe4eed333c"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.301136 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "e1329cfe5acd40808df49d3504fcfef9"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "19deed9f94c893743db435b1394c2914"
server_key_iv: "c3345eda368a7652e64c58fe4eed333c"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.302634 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:56:32.303810 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.304070 4801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:32.304190 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "e1329cfe5acd40808df49d3504fcfef9"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "19deed9f94c893743db435b1394c2914"
server_key_iv: "c3345eda368a7652e64c58fe4eed333c"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.304275 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:32.320444 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:32.322794 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:32.322926 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:32.323129 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:32.323457 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:32.323493 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.323514 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:32.323525 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.329547 4801 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:45035
I20260620 22:56:32.329730 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:45035 every 8 connection(s)
I20260620 22:56:32.329936 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:56:32.331604 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 4801
I20260620 22:56:32.331708 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:32.331910 1688 external_mini_cluster.cc:1428] Setting key 33f4c7b5bee2b95e179e1f9b1366033e
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
I20260620 22:56:32.341993 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:32.331059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:45365 (local address 127.1.166.62:38035)
0620 22:56:32.331136 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:32.331139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:32.331403 (+ 264us) server_negotiation.cc:407] Connection header received
0620 22:56:32.332169 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:32.332170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:32.332190 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:32.332245 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:32.333868 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.334607 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.335739 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.336022 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.336108 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:32.338385 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:32.338387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:32.338387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:32.338398 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:32.340125 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.340706 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.340707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.340708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.340778 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.341160 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.341161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.341162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.341334 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:32.341416 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:32.341797 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 22:56:32.341888 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:56:32.342510 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:32.331011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38035 (local address 127.1.166.1:45365)
0620 22:56:32.331216 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:32.331278 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:32.332059 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:32.332598 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:32.332604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:32.333424 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:32.333670 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:32.333681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.334718 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.334719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:32.335623 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:32.335627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.336141 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.336142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:32.336264 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:32.336850 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:32.336859 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:32.338246 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:32.340244 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:32.340254 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:32.340257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:32.340514 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:32.340868 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:32.340869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:32.340870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:32.341080 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:32.341415 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:32.341422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:32.341628 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 22:56:32.341735 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 22:56:32.341950 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
I20260620 22:56:32.343421 4930 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38035
I20260620 22:56:32.343575 4930 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:32.343820 4930 heartbeater.cc:511] Master 127.1.166.62:38035 requested a full tablet report, sending...
I20260620 22:56:32.344547 4739 ts_manager.cc:194] Registered new tserver with Master: e1329cfe5acd40808df49d3504fcfef9 (127.1.166.1:45035)
I20260620 22:56:32.345220 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:45365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:32.360869 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38035
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:32.449545 4935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:32.449753 4935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:32.449785 4935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:32.451030 4935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:32.451073 4935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:32.451110 4935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:32.453068 4935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:32.453325 4935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:32.453549 4935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:32.458312 4941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:32.458325 4943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:32.458402 4940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:32.458523 4935 server_base.cc:1034] running on GCE node
I20260620 22:56:32.458747 4935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:32.459002 4935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:32.460184 4935 hybrid_clock.cc:648] HybridClock initialized: now 1781996192460171 us; error 37 us; skew 500 ppm
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:32.462893 4935 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:32.463315 4935 webserver.cc:460] Webserver started at http://127.1.166.2:35751/ using document root <none> and password file <none>
I20260620 22:56:32.463496 4935 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:32.463538 4935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:32.463624 4935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:32.464784 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "4949036e0240413ea14c06075f8ed558"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "919711ba94799304b8338effa2b4f8e9"
server_key_iv: "c68862983c5db97b086674fdade22a4d"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.465116 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "4949036e0240413ea14c06075f8ed558"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "919711ba94799304b8338effa2b4f8e9"
server_key_iv: "c68862983c5db97b086674fdade22a4d"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.466509 4935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:56:32.467296 4950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.467475 4935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:32.467631 4935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "4949036e0240413ea14c06075f8ed558"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "919711ba94799304b8338effa2b4f8e9"
server_key_iv: "c68862983c5db97b086674fdade22a4d"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.467695 4935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:32.484654 4935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:32.486907 4935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:32.487038 4935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:32.487237 4935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:32.487509 4935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:32.487538 4935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.487560 4935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:32.487571 4935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.494059 4935 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:41973
I20260620 22:56:32.494169 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:41973 every 8 connection(s)
I20260620 22:56:32.494563 4935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:32.496853 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 4935
I20260620 22:56:32.496966 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:32.497141 1688 external_mini_cluster.cc:1428] Setting key bbbd3b90be53b92e9219a4d5889ed2c3
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
I20260620 22:56:32.506138 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:32.495436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:40525 (local address 127.1.166.62:38035)
0620 22:56:32.495528 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:32.495531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:32.495826 (+ 295us) server_negotiation.cc:407] Connection header received
0620 22:56:32.496566 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:32.496567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:32.496584 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:32.496679 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:32.497822 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.498384 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.499493 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.499833 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.499908 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:32.502910 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:32.502913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:32.502913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:32.502925 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:32.504432 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.504990 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.504992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.504992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.505065 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.505338 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.505339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.505339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.505493 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:32.505562 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:32.505900 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 22:56:32.506013 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 22:56:32.506348 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:32.495417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38035 (local address 127.1.166.2:40525)
0620 22:56:32.495646 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:32.495697 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:32.496345 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:32.496814 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:32.496820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:32.497479 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:32.497703 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:32.497709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.498472 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.498473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:32.499354 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:32.499357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.499944 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.499945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:32.500122 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:32.500760 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:32.500771 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:32.502755 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:32.504567 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:32.504582 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:32.504586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:32.504899 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:32.505135 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:32.505136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:32.505136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:32.505269 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:32.505561 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:32.505570 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:32.505755 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 22:56:32.505869 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 22:56:32.506089 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":113,"threads_started":1}
I20260620 22:56:32.507405 5064 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38035
I20260620 22:56:32.507556 5064 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:32.507787 5064 heartbeater.cc:511] Master 127.1.166.62:38035 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
I20260620 22:56:32.508381 4739 ts_manager.cc:194] Registered new tserver with Master: 4949036e0240413ea14c06075f8ed558 (127.1.166.2:41973)
I20260620 22:56:32.508904 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:40525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:32.523478 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38035
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:32.615492 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:32.615957 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:32.615998 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:32.618181 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:32.618237 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:32.618269 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:32.620303 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:32.620574 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:32.620784 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:32.625943 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:32.626076 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:32.626227 5069 server_base.cc:1034] running on GCE node
W20260620 22:56:32.626269 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:32.626539 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:32.626812 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:32.628012 5069 hybrid_clock.cc:648] HybridClock initialized: now 1781996192627989 us; error 37 us; skew 500 ppm
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:32.631421 5069 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:32.631999 5069 webserver.cc:460] Webserver started at http://127.1.166.3:36507/ using document root <none> and password file <none>
I20260620 22:56:32.632232 5069 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:32.632282 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:32.632417 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:32.633771 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "1e8ec9c9a0264571be2ece4b5a3a579c"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "636f559446177a4c279711f61d734273"
server_key_iv: "b1dcebda40a86f115f3c931b68275741"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.634200 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "1e8ec9c9a0264571be2ece4b5a3a579c"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "636f559446177a4c279711f61d734273"
server_key_iv: "b1dcebda40a86f115f3c931b68275741"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.635870 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:32.636831 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.637112 5069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:32.637212 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "1e8ec9c9a0264571be2ece4b5a3a579c"
format_stamp: "Formatted at 2026-06-20 22:56:32 on dist-test-slave-rdl9"
server_key: "636f559446177a4c279711f61d734273"
server_key_iv: "b1dcebda40a86f115f3c931b68275741"
server_key_version: "encryptionkey@0"
I20260620 22:56:32.637370 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:32.655941 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:32.658202 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:32.658367 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:32.658596 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:32.658900 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:32.658932 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.658960 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:32.658977 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:32.664760 5069 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:46051
I20260620 22:56:32.664806 5197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:46051 every 8 connection(s)
I20260620 22:56:32.665125 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:32.669721 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 5069
I20260620 22:56:32.669824 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:32.669979 1688 external_mini_cluster.cc:1428] Setting key 49457fbe6c3d50660dbd3bdc37596859
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:32.676441 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:32.666105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:54229 (local address 127.1.166.62:38035)
0620 22:56:32.666270 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:32.666273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:32.666502 (+ 229us) server_negotiation.cc:407] Connection header received
0620 22:56:32.667179 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:32.667181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:32.667195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:32.667251 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:32.668496 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.669014 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.671006 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.671226 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.671267 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:32.673767 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:32.673777 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:32.673777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:32.673785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:32.675065 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.675496 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.675497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.675498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.675553 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.675780 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.675781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.675781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.675907 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:32.675964 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:32.676246 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 22:56:32.676341 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 22:56:32.676687 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:32.666031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38035 (local address 127.1.166.3:54229)
0620 22:56:32.666299 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:32.666387 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:32.667068 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:32.667335 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:32.667341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:32.668102 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:32.668327 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:32.668336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.669114 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.669115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:32.670896 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:32.670900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.671346 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.671346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:32.671466 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:32.672040 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:32.672050 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:32.673670 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:32.675151 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:32.675159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:32.675162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:32.675414 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:32.675608 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:32.675609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:32.675609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:32.675724 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:32.675964 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:32.675971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:32.676120 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 22:56:32.676216 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 22:56:32.676444 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":126,"threads_started":1}
I20260620 22:56:32.677325 5198 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38035
I20260620 22:56:32.677433 5198 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:32.677629 5198 heartbeater.cc:511] Master 127.1.166.62:38035 requested a full tablet report, sending...
I20260620 22:56:32.678066 4739 ts_manager.cc:194] Registered new tserver with Master: 1e8ec9c9a0264571be2ece4b5a3a579c (127.1.166.3:46051)
I20260620 22:56:32.678503 4739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:54229
I20260620 22:56:32.682224 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[4694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996192, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:32.708114 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:32.699892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.1.166.62:38035)
0620 22:56:32.700008 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:32.700010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:32.700063 (+ 53us) server_negotiation.cc:407] Connection header received
0620 22:56:32.700149 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:32.700150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:32.700162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:32.700210 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:32.700831 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.701336 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.702057 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:32.702237 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:32.702291 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:32.705427 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:32.705428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:32.705429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:32.705437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:32.706836 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.707204 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.707205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.707205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.707271 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:32.707536 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:32.707538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:32.707538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:32.707710 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:32.707801 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:32.707929 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:56:32.708016 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 22:56:32.708362 4739 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42640
I20260620 22:56:32.709041 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 4801
I20260620 22:56:32.714072 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 4935
I20260620 22:56:32.719012 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 5069
I20260620 22:56:32.724292 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 4707
2026-06-20T22:56:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1847 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[5210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[5210](info): set up 2 sockets
Jun 20 22:56:32 dist-test-slave-rdl9 krb5kdc[5210](info): commencing operation
krb5kdc: starting...
W20260620 22:56:33.698187 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:33 dist-test-slave-rdl9 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:33.817121 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:42053
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:42053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:33.905046 5223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:33.905373 5223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:33.905414 5223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:33.906723 5223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:33.906767 5223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:33.906780 5223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:33.906793 5223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:33.906805 5223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:33.908481 5223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:42053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:42053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:33.908740 5223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:33.908984 5223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:33.913950 5231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:33.913913 5228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:33.913913 5229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:33.914415 5223 server_base.cc:1034] running on GCE node
I20260620 22:56:33.914685 5223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:33.914991 5223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:33.916205 5223 hybrid_clock.cc:648] HybridClock initialized: now 1781996193916188 us; error 32 us; skew 500 ppm
Jun 20 22:56:33 dist-test-slave-rdl9 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996193, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:33.919343 5223 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:33.919894 5223 webserver.cc:460] Webserver started at http://127.1.166.62:42141/ using document root <none> and password file <none>
I20260620 22:56:33.920136 5223 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:33.920185 5223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:33.920303 5223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:33.921432 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "2c07be5b3e5c4061b21f8eadec275bf5"
format_stamp: "Formatted at 2026-06-20 22:56:33 on dist-test-slave-rdl9"
server_key: "161afafb49551e3f59e015fe4f9baab2"
server_key_iv: "9c6f19b8663cd65273f3ba4541f418c4"
server_key_version: "encryptionkey@0"
I20260620 22:56:33.921782 5223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "2c07be5b3e5c4061b21f8eadec275bf5"
format_stamp: "Formatted at 2026-06-20 22:56:33 on dist-test-slave-rdl9"
server_key: "161afafb49551e3f59e015fe4f9baab2"
server_key_iv: "9c6f19b8663cd65273f3ba4541f418c4"
server_key_version: "encryptionkey@0"
I20260620 22:56:33.923399 5223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:56:33.924381 5238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:33.924741 5223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:33.924880 5223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "2c07be5b3e5c4061b21f8eadec275bf5"
format_stamp: "Formatted at 2026-06-20 22:56:33 on dist-test-slave-rdl9"
server_key: "161afafb49551e3f59e015fe4f9baab2"
server_key_iv: "9c6f19b8663cd65273f3ba4541f418c4"
server_key_version: "encryptionkey@0"
I20260620 22:56:33.924997 5223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:33.947026 5223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:33.950062 5223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:33.950325 5223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:33.954985 5223 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:42053
I20260620 22:56:33.955070 5290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:42053 every 8 connection(s)
I20260620 22:56:33.955525 5223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:33.956209 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:33.958458 5291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5: Bootstrap starting.
I20260620 22:56:33.959019 5291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:33.959286 5291 log.cc:824] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:33.960258 5291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5: No bootstrap required, opened a new log
I20260620 22:56:33.962163 5291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c07be5b3e5c4061b21f8eadec275bf5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 42053 } }
I20260620 22:56:33.962378 5291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:33.962430 5291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c07be5b3e5c4061b21f8eadec275bf5, State: Initialized, Role: FOLLOWER
I20260620 22:56:33.962561 5291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c07be5b3e5c4061b21f8eadec275bf5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 42053 } }
I20260620 22:56:33.962638 5291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:33.962670 5291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:33.962704 5291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:33.963430 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 5223
I20260620 22:56:33.963539 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:33.963708 1688 external_mini_cluster.cc:1428] Setting key 3c30d0d1637f341573ca3fd465b18098
I20260620 22:56:33.963825 5291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c07be5b3e5c4061b21f8eadec275bf5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 42053 } }
I20260620 22:56:33.963934 5291 leader_election.cc:302] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c07be5b3e5c4061b21f8eadec275bf5; no voters:
I20260620 22:56:33.964103 5291 leader_election.cc:288] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:33.964145 5294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:33.964416 5294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [term 1 LEADER]: Becoming Leader. State: Replica: 2c07be5b3e5c4061b21f8eadec275bf5, State: Running, Role: LEADER
I20260620 22:56:33.964506 5291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:33.964534 5294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c07be5b3e5c4061b21f8eadec275bf5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 42053 } }
I20260620 22:56:33.964872 5297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c07be5b3e5c4061b21f8eadec275bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c07be5b3e5c4061b21f8eadec275bf5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 42053 } } }
I20260620 22:56:33.964962 5297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:33.964872 5298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c07be5b3e5c4061b21f8eadec275bf5. Latest consensus state: current_term: 1 leader_uuid: "2c07be5b3e5c4061b21f8eadec275bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c07be5b3e5c4061b21f8eadec275bf5" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 42053 } } }
I20260620 22:56:33.965250 5298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:33.965394 5302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:33.965880 5302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:33.968230 5302 catalog_manager.cc:1347] Generated new cluster ID: be094cb85b4c416ab3858ac87a1ffbd6
I20260620 22:56:33.968318 5302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:56:33 dist-test-slave-rdl9 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:33.973959 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:33.964194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.1.166.62:42053)
0620 22:56:33.964534 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:33.964587 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:33.964620 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:56:33.964749 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:33.964758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:33.964817 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:33.964965 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:33.965734 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:33.967113 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:33.968177 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:33.968693 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:33.968807 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:33.971058 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:33.971062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:33.971062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:33.971077 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:33.972578 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:33.972890 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:33.972893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:33.972894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:33.972982 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:33.973150 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:33.973151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:33.973151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:33.973393 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:33.973557 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:33.973624 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 22:56:33.973761 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":75,"threads_started":1}
I20260620 22:56:33.980829 5302 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:33.981572 5302 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:33.990262 5302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c07be5b3e5c4061b21f8eadec275bf5: Generated new TSK 0
I20260620 22:56:33.990548 5302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:34.020954 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:42053
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:34.113379 5317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:34.113646 5317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:34.113687 5317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:34.115244 5317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:34.115314 5317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:34.115358 5317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:34.117216 5317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:42053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:34.117501 5317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:34.117741 5317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:34.122771 5325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:34.122771 5323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:34.122859 5322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:34.122941 5317 server_base.cc:1034] running on GCE node
I20260620 22:56:34.123160 5317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:34.123409 5317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:34.124588 5317 hybrid_clock.cc:648] HybridClock initialized: now 1781996194124574 us; error 29 us; skew 500 ppm
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:34.127912 5317 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:34.128461 5317 webserver.cc:460] Webserver started at http://127.1.166.1:45833/ using document root <none> and password file <none>
I20260620 22:56:34.128685 5317 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:34.128732 5317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:34.128870 5317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:34.130034 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "9ec803de68f64b96a7dd1709d3fe5482"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "1d2f2f0504f14167c92e21995797979b"
server_key_iv: "d8e3c6c1eddd83963ed7eed4dadb2948"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.130437 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "9ec803de68f64b96a7dd1709d3fe5482"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "1d2f2f0504f14167c92e21995797979b"
server_key_iv: "d8e3c6c1eddd83963ed7eed4dadb2948"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.132510 5317 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 22:56:34.133700 5332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.133955 5317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:34.134074 5317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "9ec803de68f64b96a7dd1709d3fe5482"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "1d2f2f0504f14167c92e21995797979b"
server_key_iv: "d8e3c6c1eddd83963ed7eed4dadb2948"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.134195 5317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:34.159785 5317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:34.161966 5317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:34.162091 5317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:34.162328 5317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:34.162653 5317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:34.162684 5317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.162710 5317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:34.162729 5317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.169093 5317 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:38413
I20260620 22:56:34.169162 5445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:38413 every 8 connection(s)
I20260620 22:56:34.169795 5317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:34.177230 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 5317
I20260620 22:56:34.177343 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:34.177520 1688 external_mini_cluster.cc:1428] Setting key 3705052f2edb6b4de3040bb37dbdbdb1
I20260620 22:56:34.180516 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:34.170748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:57711 (local address 127.1.166.62:42053)
0620 22:56:34.170800 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:34.170802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:34.171005 (+ 203us) server_negotiation.cc:407] Connection header received
0620 22:56:34.171914 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:34.171915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:34.171931 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:34.171993 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:34.172625 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.173211 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.174401 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.174622 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.174690 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:34.177568 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:34.177570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:34.177570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:34.177583 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:34.179148 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.179601 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.179602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.179602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.179658 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.179888 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.179889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.179890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.180001 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:34.180059 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:34.180331 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 22:56:34.180422 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:56:34.180722 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:34.170674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:42053 (local address 127.1.166.1:57711)
0620 22:56:34.170854 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:34.170897 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:34.171803 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:34.172060 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:34.172064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:34.172322 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:34.172525 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:34.172529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.173319 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.173320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:34.174263 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:34.174266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.174774 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.174776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:34.174956 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:34.175617 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:34.175629 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:34.177450 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:34.179238 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:34.179245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:34.179247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:34.179510 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:34.179715 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:34.179715 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:34.179715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:34.179831 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:34.180055 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:34.180061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:34.180214 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 22:56:34.180305 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 22:56:34.180494 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260620 22:56:34.181222 5446 heartbeater.cc:347] Connected to a master server at 127.1.166.62:42053
I20260620 22:56:34.181340 5446 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:34.181542 5446 heartbeater.cc:511] Master 127.1.166.62:42053 requested a full tablet report, sending...
I20260620 22:56:34.182065 5255 ts_manager.cc:194] Registered new tserver with Master: 9ec803de68f64b96a7dd1709d3fe5482 (127.1.166.1:38413)
I20260620 22:56:34.182718 5255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:57711
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:34.206241 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:42053
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:34.291325 5451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:34.291563 5451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:34.291590 5451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:34.292848 5451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:34.292903 5451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:34.292932 5451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:34.294677 5451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:42053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:34.294931 5451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:34.295145 5451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:34.300128 5456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:34.300128 5457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:34.300128 5459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:34.300457 5451 server_base.cc:1034] running on GCE node
I20260620 22:56:34.300666 5451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:34.300889 5451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:34.302024 5451 hybrid_clock.cc:648] HybridClock initialized: now 1781996194302005 us; error 40 us; skew 500 ppm
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:34.304738 5451 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:34.305343 5451 webserver.cc:460] Webserver started at http://127.1.166.2:44791/ using document root <none> and password file <none>
I20260620 22:56:34.305552 5451 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:34.305586 5451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:34.305673 5451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:34.306770 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "e37487740d084c05a852730ce6921ecf"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "6ae709d9872f389fb1bf10e35ef9fe2b"
server_key_iv: "3df90f562e38c2ee990e72cb08a9090c"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.307104 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "e37487740d084c05a852730ce6921ecf"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "6ae709d9872f389fb1bf10e35ef9fe2b"
server_key_iv: "3df90f562e38c2ee990e72cb08a9090c"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.308673 5451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:34.309629 5466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.309842 5451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:34.309943 5451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "e37487740d084c05a852730ce6921ecf"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "6ae709d9872f389fb1bf10e35ef9fe2b"
server_key_iv: "3df90f562e38c2ee990e72cb08a9090c"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.310011 5451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:34.321765 5451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:34.324177 5451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:34.324340 5451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:34.324568 5451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:34.324893 5451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:34.324928 5451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.324958 5451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:34.324982 5451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.331578 5451 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:39317
I20260620 22:56:34.331634 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:39317 every 8 connection(s)
I20260620 22:56:34.331982 5451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:34.341765 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 5451
I20260620 22:56:34.341877 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:34.342069 1688 external_mini_cluster.cc:1428] Setting key 40cd23f3ad0512b59b953ac974d3d401
I20260620 22:56:34.343217 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:34.333076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:56351 (local address 127.1.166.62:42053)
0620 22:56:34.333218 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:34.333221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:34.333436 (+ 215us) server_negotiation.cc:407] Connection header received
0620 22:56:34.334648 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:34.334650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:34.334665 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:34.334716 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:34.335755 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.336292 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.337345 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.337554 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.337614 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:34.340112 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:34.340114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:34.340114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:34.340122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:34.341500 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.341974 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.341975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.341975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.342045 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.342441 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.342443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.342443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.342592 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:34.342654 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:34.343032 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 22:56:34.343133 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 22:56:34.343354 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:34.332920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:42053 (local address 127.1.166.2:56351)
0620 22:56:34.333281 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:34.333335 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:34.334532 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:34.334799 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:34.334803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:34.335331 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:34.335580 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:34.335588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.336402 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.336403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:34.337243 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:34.337245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.337625 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.337626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:34.337763 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:34.338394 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:34.338403 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:34.339992 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:34.341606 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:34.341614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:34.341616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:34.341874 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:34.342145 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:34.342146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:34.342146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:34.342352 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:34.342658 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:34.342665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:34.342861 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 22:56:34.342966 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 22:56:34.343120 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":187,"threads_started":1}
I20260620 22:56:34.343957 5580 heartbeater.cc:347] Connected to a master server at 127.1.166.62:42053
I20260620 22:56:34.344071 5580 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:34.344293 5580 heartbeater.cc:511] Master 127.1.166.62:42053 requested a full tablet report, sending...
I20260620 22:56:34.344755 5255 ts_manager.cc:194] Registered new tserver with Master: e37487740d084c05a852730ce6921ecf (127.1.166.2:39317)
I20260620 22:56:34.345247 5255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:56351
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:34.369333 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:42053
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:34.463140 5585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:34.463372 5585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:34.463399 5585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:34.465111 5585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:34.465199 5585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:34.465250 5585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:34.468393 5585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:42053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:34.468721 5585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:34.468956 5585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:34.474001 5591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:34.473997 5593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:34.474370 5585 server_base.cc:1034] running on GCE node
W20260620 22:56:34.474457 5590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:34.474766 5585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:34.475041 5585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:34.476248 5585 hybrid_clock.cc:648] HybridClock initialized: now 1781996194476218 us; error 39 us; skew 500 ppm
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:34.479724 5585 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:34.480362 5585 webserver.cc:460] Webserver started at http://127.1.166.3:35587/ using document root <none> and password file <none>
I20260620 22:56:34.480595 5585 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:34.480650 5585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:34.480796 5585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:34.481964 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "c8a6fdbb07414c8bb4ff0e483bb4e1a6"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "937446da47ca877306bf7650a7a5336a"
server_key_iv: "d36b11a3b7c3b963ad4a9a3b54fc45b5"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.482316 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "c8a6fdbb07414c8bb4ff0e483bb4e1a6"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "937446da47ca877306bf7650a7a5336a"
server_key_iv: "d36b11a3b7c3b963ad4a9a3b54fc45b5"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.483856 5585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:34.485069 5600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.485332 5585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:34.485458 5585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "c8a6fdbb07414c8bb4ff0e483bb4e1a6"
format_stamp: "Formatted at 2026-06-20 22:56:34 on dist-test-slave-rdl9"
server_key: "937446da47ca877306bf7650a7a5336a"
server_key_iv: "d36b11a3b7c3b963ad4a9a3b54fc45b5"
server_key_version: "encryptionkey@0"
I20260620 22:56:34.485564 5585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:34.511740 5585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:34.514014 5585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:34.514137 5585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:34.514384 5585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:34.514675 5585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:34.514704 5585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.514731 5585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:34.514748 5585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:34.520864 5585 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:35337
I20260620 22:56:34.520941 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:35337 every 8 connection(s)
I20260620 22:56:34.521250 5585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:34.526300 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 5585
I20260620 22:56:34.526461 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:34.526660 1688 external_mini_cluster.cc:1428] Setting key b95e6cf06de0ad592c955c7a8d8f1940
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:34.533821 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:34.522225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:41937 (local address 127.1.166.62:42053)
0620 22:56:34.522361 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:34.522364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:34.522567 (+ 203us) server_negotiation.cc:407] Connection header received
0620 22:56:34.523735 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:34.523737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:34.523752 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:34.523848 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:34.524806 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.525434 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.526976 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.527227 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.527299 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:34.530322 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:34.530324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:34.530324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:34.530369 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:34.531968 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.532511 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.532513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.532513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.532577 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.532971 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.532973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.532973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.533123 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:34.533202 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:34.533650 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 22:56:34.533743 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 22:56:34.533994 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:34.522071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:42053 (local address 127.1.166.3:41937)
0620 22:56:34.522319 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:34.522429 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:34.523593 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:34.523918 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:34.523924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:34.524273 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:34.524541 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:34.524545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.525593 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.525595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:34.526873 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:34.526876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.527385 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.527388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:34.527634 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:34.528299 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:34.528308 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:34.530044 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:34.532083 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:34.532094 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:34.532097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:34.532420 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:34.532690 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:34.532692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:34.532693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:34.532866 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:34.533237 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:34.533244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:34.533420 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 22:56:34.533566 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 22:56:34.533745 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
I20260620 22:56:34.534700 5714 heartbeater.cc:347] Connected to a master server at 127.1.166.62:42053
I20260620 22:56:34.534828 5714 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:34.535044 5714 heartbeater.cc:511] Master 127.1.166.62:42053 requested a full tablet report, sending...
I20260620 22:56:34.535517 5255 ts_manager.cc:194] Registered new tserver with Master: c8a6fdbb07414c8bb4ff0e483bb4e1a6 (127.1.166.3:35337)
I20260620 22:56:34.535887 5255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:41937
I20260620 22:56:34.539119 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:34.563727 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:34.554146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.1.166.1:38413)
0620 22:56:34.554300 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:34.554305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:34.554313 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:34.554355 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:34.554358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:34.554383 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:34.554451 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:34.555049 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.555788 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.556870 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.557318 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.557399 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:34.560080 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:34.560094 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:34.560095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:34.560105 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:34.562122 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.562800 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.562806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.562806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.562885 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.563065 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.563066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.563066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.563259 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:34.563442 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:34.563491 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 22:56:34.563597 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:34.573860 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:34.565614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.1.166.62:42053)
0620 22:56:34.565657 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:34.565659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:34.565669 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:34.565725 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:34.565726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:34.565735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:34.565810 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:34.566473 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.566995 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.567887 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:34.568118 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:34.568185 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:34.570949 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:34.570951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:34.570951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:34.571037 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:34.572574 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.572924 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.572925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.572926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.572989 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:34.573150 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:34.573152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:34.573152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:34.573361 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:34.573516 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:34.573651 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:56:34.573749 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:56:34.574265 5255 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44784
I20260620 22:56:34.574745 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 5317
I20260620 22:56:34.580482 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 5451
I20260620 22:56:34.584666 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 5585
I20260620 22:56:34.589205 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 5223
2026-06-20T22:56:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1864 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5726](info): set up 2 sockets
Jun 20 22:56:34 dist-test-slave-rdl9 krb5kdc[5726](info): commencing operation
krb5kdc: starting...
W20260620 22:56:35.562103 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:35 dist-test-slave-rdl9 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:35.684976 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38803
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:38803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:35.774133 5739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:35.774433 5739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:35.774472 5739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:35.775776 5739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:35.775840 5739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:35.775859 5739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:35.775883 5739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:35.775898 5739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:35.778146 5739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:38803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:35.778759 5739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:35.779127 5739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:35.784886 5747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:35.784909 5745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:35.784886 5744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:35.785172 5739 server_base.cc:1034] running on GCE node
I20260620 22:56:35.785421 5739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:35.785737 5739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:35.786919 5739 hybrid_clock.cc:648] HybridClock initialized: now 1781996195786905 us; error 34 us; skew 500 ppm
Jun 20 22:56:35 dist-test-slave-rdl9 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:35.789451 5739 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:35.789801 5739 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:35.789855 5739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:35.789987 5739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:35.791121 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "fca3417b80ea468095ab838da4124ff7"
format_stamp: "Formatted at 2026-06-20 22:56:35 on dist-test-slave-rdl9"
server_key: "e78bb752122d2772c2093e6ab61ec4fb"
server_key_iv: "7e9b9ed6b6e015a38a55aee924bcdc9a"
server_key_version: "encryptionkey@0"
I20260620 22:56:35.791445 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "fca3417b80ea468095ab838da4124ff7"
format_stamp: "Formatted at 2026-06-20 22:56:35 on dist-test-slave-rdl9"
server_key: "e78bb752122d2772c2093e6ab61ec4fb"
server_key_iv: "7e9b9ed6b6e015a38a55aee924bcdc9a"
server_key_version: "encryptionkey@0"
I20260620 22:56:35.792793 5739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:56:35.793706 5753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:35.793900 5739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:35.793988 5739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "fca3417b80ea468095ab838da4124ff7"
format_stamp: "Formatted at 2026-06-20 22:56:35 on dist-test-slave-rdl9"
server_key: "e78bb752122d2772c2093e6ab61ec4fb"
server_key_iv: "7e9b9ed6b6e015a38a55aee924bcdc9a"
server_key_version: "encryptionkey@0"
I20260620 22:56:35.794057 5739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:35.811480 5739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:35.813880 5739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:35.814046 5739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:35.818186 5739 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:38803
I20260620 22:56:35.818231 5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:38803 every 8 connection(s)
I20260620 22:56:35.818570 5739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:35.819641 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:35.821055 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 5739
I20260620 22:56:35.821162 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:35.821338 1688 external_mini_cluster.cc:1428] Setting key cda19d7838070d58e82314409c34eed1
I20260620 22:56:35.822253 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7: Bootstrap starting.
I20260620 22:56:35.822885 5806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:35.823205 5806 log.cc:824] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:35.824046 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7: No bootstrap required, opened a new log
I20260620 22:56:35.826375 5806 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca3417b80ea468095ab838da4124ff7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38803 } }
I20260620 22:56:35.826555 5806 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:35.826581 5806 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca3417b80ea468095ab838da4124ff7, State: Initialized, Role: FOLLOWER
I20260620 22:56:35.826723 5806 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca3417b80ea468095ab838da4124ff7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38803 } }
I20260620 22:56:35.826810 5806 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:35.826843 5806 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:35.826898 5806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:35.827627 5806 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca3417b80ea468095ab838da4124ff7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38803 } }
I20260620 22:56:35.827749 5806 leader_election.cc:302] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca3417b80ea468095ab838da4124ff7; no voters:
I20260620 22:56:35.827934 5806 leader_election.cc:288] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:35.828033 5811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:35.828187 5811 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [term 1 LEADER]: Becoming Leader. State: Replica: fca3417b80ea468095ab838da4124ff7, State: Running, Role: LEADER
I20260620 22:56:35.828212 5806 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:35.828274 5811 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca3417b80ea468095ab838da4124ff7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38803 } }
I20260620 22:56:35.828616 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca3417b80ea468095ab838da4124ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca3417b80ea468095ab838da4124ff7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38803 } } }
I20260620 22:56:35.828682 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [sys.catalog]: This master's current role is: LEADER
Jun 20 22:56:35 dist-test-slave-rdl9 krb5kdc[5726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:35.828830 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca3417b80ea468095ab838da4124ff7. Latest consensus state: current_term: 1 leader_uuid: "fca3417b80ea468095ab838da4124ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca3417b80ea468095ab838da4124ff7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38803 } } }
I20260620 22:56:35.828874 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:35.828995 5818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:35.829489 5818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:35.831904 5818 catalog_manager.cc:1347] Generated new cluster ID: 5117fdb247c94a069a9f3fd2e7fa97a4
I20260620 22:56:35.831960 5818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:35.833173 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:35.821959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.1.166.62:38803)
0620 22:56:35.822103 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:35.822110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:35.822134 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:56:35.822590 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:35.822597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:35.822647 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:35.822768 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:35.823500 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:35.824631 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:35.825767 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:35.826407 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:35.826687 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:35.829527 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:35.829532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:35.829532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:35.829546 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:35.831478 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:35.831887 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:35.831890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:35.831891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:35.831990 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:35.832209 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:35.832210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:35.832210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:35.832550 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:35.832692 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:35.832802 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 22:56:35.832957 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260620 22:56:35.844276 5818 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:35.844858 5818 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:35.859411 5818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7: Generated new TSK 0
I20260620 22:56:35.859808 5818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:35.891561 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38803
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:35.983857 5832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:35.984161 5832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:35.984212 5832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:35.986527 5832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:35.986593 5832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:35.986637 5832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:35.988624 5832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:35.988932 5832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:35.989185 5832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:35.994244 5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:35.994246 5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:35.994244 5838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:35.994673 5832 server_base.cc:1034] running on GCE node
I20260620 22:56:35.994915 5832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:35.995263 5832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:35.996441 5832 hybrid_clock.cc:648] HybridClock initialized: now 1781996195996428 us; error 32 us; skew 500 ppm
Jun 20 22:56:35 dist-test-slave-rdl9 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:35.999804 5832 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:36.000348 5832 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:36.000416 5832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:36.000586 5832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:36.001853 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "168b65e2251643a980cb537232577be9"
format_stamp: "Formatted at 2026-06-20 22:56:35 on dist-test-slave-rdl9"
server_key: "f7d0c4f08459ad50ba4f430612fd491b"
server_key_iv: "619e6840c04c17e4a7ba99359281780e"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.002450 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "168b65e2251643a980cb537232577be9"
format_stamp: "Formatted at 2026-06-20 22:56:35 on dist-test-slave-rdl9"
server_key: "f7d0c4f08459ad50ba4f430612fd491b"
server_key_iv: "619e6840c04c17e4a7ba99359281780e"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.004331 5832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:56:36.005448 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.005769 5832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:36.005868 5832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "168b65e2251643a980cb537232577be9"
format_stamp: "Formatted at 2026-06-20 22:56:35 on dist-test-slave-rdl9"
server_key: "f7d0c4f08459ad50ba4f430612fd491b"
server_key_iv: "619e6840c04c17e4a7ba99359281780e"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.005956 5832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:36.014385 5832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:36.016698 5832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:36.016851 5832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:36.017108 5832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:36.017437 5832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:36.017478 5832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.017509 5832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:36.017526 5832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.023914 5832 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:45967
I20260620 22:56:36.024011 5959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:45967 every 8 connection(s)
I20260620 22:56:36.024288 5832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:56:36.028712 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 5832
I20260620 22:56:36.028837 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:36.029026 1688 external_mini_cluster.cc:1428] Setting key ddfaeedaae73877a9065692c38d76331
Jun 20 22:56:36 dist-test-slave-rdl9 krb5kdc[5726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:36.036051 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.025056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:33417 (local address 127.1.166.62:38803)
0620 22:56:36.025134 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.025136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.025440 (+ 304us) server_negotiation.cc:407] Connection header received
0620 22:56:36.025951 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.025952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.025963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.026016 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:36.027167 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.027790 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.029409 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.029660 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.029727 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.032182 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:36.032185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:36.032185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:36.032201 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:36.034095 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:36.034757 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:36.034759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:36.034760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:36.034863 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:36.035193 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:36.035194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:36.035194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:36.035371 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:36.035485 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.035814 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.035945 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:56:36.036099 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.025052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38803 (local address 127.1.166.1:33417)
0620 22:56:36.025291 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.025334 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.025857 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.026088 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.026096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.026462 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:36.027027 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.027033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.027924 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.027925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.029325 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.029328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.029746 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.029748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.029893 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.030454 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:36.030462 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:36.031978 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:36.034228 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:36.034238 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:36.034240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:36.034588 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:36.034979 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:36.034980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:36.034980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:36.035110 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:36.035495 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:36.035501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:36.035658 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.035756 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.035903 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260620 22:56:36.036856 5960 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38803
I20260620 22:56:36.037000 5960 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:36.037159 5960 heartbeater.cc:511] Master 127.1.166.62:38803 requested a full tablet report, sending...
I20260620 22:56:36.037781 5770 ts_manager.cc:194] Registered new tserver with Master: 168b65e2251643a980cb537232577be9 (127.1.166.1:45967)
I20260620 22:56:36.038484 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:33417
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:36.058032 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38803
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:36.146395 5965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:36.146651 5965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:36.146677 5965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:36.148197 5965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:36.148270 5965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:36.148305 5965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:36.150133 5965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:36.150442 5965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:36.150686 5965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:36.156016 5973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.156059 5970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.156016 5971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:36.156347 5965 server_base.cc:1034] running on GCE node
I20260620 22:56:36.156579 5965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:36.157024 5965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:36.158205 5965 hybrid_clock.cc:648] HybridClock initialized: now 1781996196158183 us; error 38 us; skew 500 ppm
Jun 20 22:56:36 dist-test-slave-rdl9 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:36.160880 5965 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:36.161258 5965 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:36.161309 5965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:36.161427 5965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:36.162560 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "1ee664cd23d3401b81bbea2e5977ea3f"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "3eb24984a91beda86bbe72223e28202d"
server_key_iv: "96c91f27557cf5cd1767a4c29ad8c1fd"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.162899 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "1ee664cd23d3401b81bbea2e5977ea3f"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "3eb24984a91beda86bbe72223e28202d"
server_key_iv: "96c91f27557cf5cd1767a4c29ad8c1fd"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.164283 5965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:36.165182 5979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.165357 5965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:36.165442 5965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "1ee664cd23d3401b81bbea2e5977ea3f"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "3eb24984a91beda86bbe72223e28202d"
server_key_iv: "96c91f27557cf5cd1767a4c29ad8c1fd"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.165508 5965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:36.175897 5965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:36.178085 5965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:36.178205 5965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:36.178448 5965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:36.178725 5965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:36.178753 5965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.178774 5965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:36.178786 5965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.184422 5965 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:45885
I20260620 22:56:36.184463 6092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:45885 every 8 connection(s)
I20260620 22:56:36.184742 5965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:56:36 dist-test-slave-rdl9 krb5kdc[5726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:36.194252 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 5965
I20260620 22:56:36.194403 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:36.194322 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.185556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:37885 (local address 127.1.166.62:38803)
0620 22:56:36.185730 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.185732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.185935 (+ 203us) server_negotiation.cc:407] Connection header received
0620 22:56:36.186529 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.186531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.186543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.186593 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:36.187576 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.188079 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.189068 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.189256 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.189312 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.191518 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:36.191519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:36.191519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:36.191527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:36.192808 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:36.193250 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:36.193251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:36.193252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:36.193305 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:36.193642 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:36.193643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:36.193643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:36.193773 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:36.193834 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.194166 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.194249 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 22:56:36.194612 1688 external_mini_cluster.cc:1428] Setting key 149863ae8331c7824194580814020a07
I20260620 22:56:36.194664 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.185451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38803 (local address 127.1.166.2:37885)
0620 22:56:36.185800 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.185847 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.186434 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.186670 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.186675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.186966 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:36.187478 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.187483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.188170 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.188170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.188984 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.188987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.189328 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.189329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.189461 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.189985 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:36.189994 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:36.191417 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:36.192916 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:36.192934 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:36.192937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:36.193177 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:36.193360 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:36.193361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:36.193361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:36.193548 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:36.193837 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:36.193843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:36.193994 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.194111 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.194306 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":205,"threads_started":1}
I20260620 22:56:36.195401 6093 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38803
I20260620 22:56:36.195617 6093 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:36.195861 6093 heartbeater.cc:511] Master 127.1.166.62:38803 requested a full tablet report, sending...
I20260620 22:56:36.196465 5770 ts_manager.cc:194] Registered new tserver with Master: 1ee664cd23d3401b81bbea2e5977ea3f (127.1.166.2:45885)
I20260620 22:56:36.196878 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:37885
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:36.225457 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38803
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:36.316586 6098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:36.316875 6098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:36.316905 6098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:36.318379 6098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:36.318449 6098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:36.318482 6098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:36.320274 6098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:36.320561 6098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:36.320822 6098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:36.326395 6104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.326328 6103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.326443 6106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:36.326713 6098 server_base.cc:1034] running on GCE node
I20260620 22:56:36.326970 6098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:36.327258 6098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:36.328398 6098 hybrid_clock.cc:648] HybridClock initialized: now 1781996196328375 us; error 36 us; skew 500 ppm
Jun 20 22:56:36 dist-test-slave-rdl9 krb5kdc[5726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:36.331449 6098 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:36.331862 6098 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:36.331928 6098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:36.332065 6098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:36.333176 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "16cf613e8a9d4b45ac5a54d17f7bc256"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "c36d21a545157efb2154dde07c2bd957"
server_key_iv: "7c0876fc7d33dce96b5bf8b6dc67b515"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.333530 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "16cf613e8a9d4b45ac5a54d17f7bc256"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "c36d21a545157efb2154dde07c2bd957"
server_key_iv: "7c0876fc7d33dce96b5bf8b6dc67b515"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.334995 6098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:56:36.335966 6112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.336221 6098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:36.336320 6098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "16cf613e8a9d4b45ac5a54d17f7bc256"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "c36d21a545157efb2154dde07c2bd957"
server_key_iv: "7c0876fc7d33dce96b5bf8b6dc67b515"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.336403 6098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:36.356547 6098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:36.359614 6098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:36.359805 6098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:36.360102 6098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:36.360517 6098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:36.360554 6098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.360594 6098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:36.360618 6098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.368144 6098 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:46021
I20260620 22:56:36.368213 6225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:46021 every 8 connection(s)
I20260620 22:56:36.368570 6098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:36.371603 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 6098
I20260620 22:56:36.371702 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:36.371874 1688 external_mini_cluster.cc:1428] Setting key e9470b8f6f3f54d10b7ef7ca5601f37d
Jun 20 22:56:36 dist-test-slave-rdl9 krb5kdc[5726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:36.381199 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.369750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:35597 (local address 127.1.166.62:38803)
0620 22:56:36.369849 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.369852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.370095 (+ 243us) server_negotiation.cc:407] Connection header received
0620 22:56:36.370819 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.370821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.370840 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.370909 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:36.372269 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.372962 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.374128 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.374495 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.374561 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.377647 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:36.377649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:36.377649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:36.377661 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:36.379390 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:36.380008 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:36.380009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:36.380010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:36.380097 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:36.380353 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:36.380353 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:36.380353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:36.380485 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:36.380550 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.381014 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.381114 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 22:56:36.381362 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.369671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38803 (local address 127.1.166.3:35597)
0620 22:56:36.369902 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.369954 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.370660 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.371081 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.371087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.371431 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:36.372161 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.372171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.373087 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.373088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.374006 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.374010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.374663 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.374665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.374866 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.375671 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:36.375683 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:36.377470 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:36.379511 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:36.379528 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:36.379532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:36.379875 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:36.380173 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:36.380174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:36.380174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:36.380296 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:36.380615 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:36.380630 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:36.380821 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.380951 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.381133 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":116,"threads_started":1}
I20260620 22:56:36.381947 6226 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38803
I20260620 22:56:36.382074 6226 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:36.382251 6226 heartbeater.cc:511] Master 127.1.166.62:38803 requested a full tablet report, sending...
I20260620 22:56:36.382887 5770 ts_manager.cc:194] Registered new tserver with Master: 16cf613e8a9d4b45ac5a54d17f7bc256 (127.1.166.3:46021)
I20260620 22:56:36.383433 5770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:35597
I20260620 22:56:36.383667 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:56:36.392751 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.385106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.1.166.62:38803)
0620 22:56:36.385156 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.385158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.385219 (+ 61us) server_negotiation.cc:407] Connection header received
0620 22:56:36.385283 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.385283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.385292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.385339 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:36.386326 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.386960 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.387851 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.388081 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.388161 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.389606 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:36.389610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:36.389612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:36.389625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:36.391233 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:36.391725 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:36.391727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:36.391727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:36.391855 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:36.392146 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:36.392148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:36.392148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:36.392362 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:36.392466 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.392597 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.392677 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:36.394847 5770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:36.396965 5770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:36.404407 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.399961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:45967 (local address 127.0.0.1:50100)
0620 22:56:36.400319 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.400364 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.400442 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.400838 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.400842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.400847 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:36.400957 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.400960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.402326 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.402327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.403337 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.403340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.404136 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.404137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.404207 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.404212 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.404242 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.404290 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":26,"thread_start_us":89,"threads_started":1}
I20260620 22:56:36.404421 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.400475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.1.166.2:45885)
0620 22:56:36.400637 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.400642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.400652 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:36.400692 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.400694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.400710 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.400779 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:36.401017 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.401737 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.403221 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.403975 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.404053 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.404117 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.404253 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.404328 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260620 22:56:36.404542 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.400314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.1.166.1:45967)
0620 22:56:36.400669 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.400672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.400681 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:36.400714 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.400716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.400729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.401022 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:36.401095 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.402215 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.403393 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.404048 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.404339 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.404402 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.404427 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.404467 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":104,"threads_started":1}
I20260620 22:56:36.404724 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.400264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45885 (local address 127.0.0.1:39866)
0620 22:56:36.400497 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.400507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.400549 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.400867 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.400868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.400870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:36.400941 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.400943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.401826 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.401827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.403015 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.403018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.404061 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.404062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.404139 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.404156 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.404225 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.404285 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":64,"threads_started":1}
I20260620 22:56:36.405429 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.400070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:46021 (local address 127.0.0.1:33448)
0620 22:56:36.400465 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.400488 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.400543 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.400841 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.400842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.400844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:36.400957 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.400959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.403406 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.403406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.404071 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.404072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.405195 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.405196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.405259 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.405262 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.405313 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.405352 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":6,"thread_start_us":98,"threads_started":1}
I20260620 22:56:36.405443 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.400192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.1.166.3:46021)
0620 22:56:36.400654 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.400658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.400669 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:36.400709 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.400711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.400728 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.401030 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:36.401100 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.402217 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.404160 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.405122 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.405187 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.405238 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.405327 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.405368 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":91,"threads_started":1}
I20260620 22:56:36.405349 5894 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd086e44f254e818d5d1ab77b4f9240 (DEFAULT_TABLE table=test-table [id=a9de9847a24e492db60cd150cd989826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:36.405436 6027 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd086e44f254e818d5d1ab77b4f9240 (DEFAULT_TABLE table=test-table [id=a9de9847a24e492db60cd150cd989826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:36.405717 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd086e44f254e818d5d1ab77b4f9240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:36.405717 6027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd086e44f254e818d5d1ab77b4f9240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:36.409510 6244 tablet_bootstrap.cc:492] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9: Bootstrap starting.
I20260620 22:56:36.410039 6160 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd086e44f254e818d5d1ab77b4f9240 (DEFAULT_TABLE table=test-table [id=a9de9847a24e492db60cd150cd989826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:36.410328 6160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd086e44f254e818d5d1ab77b4f9240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:36.410498 6244 tablet_bootstrap.cc:654] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:36.410602 6243 tablet_bootstrap.cc:492] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Bootstrap starting.
I20260620 22:56:36.411242 6244 log.cc:824] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:36.411365 6243 tablet_bootstrap.cc:654] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:36.411912 6243 log.cc:824] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:36.412325 6244 tablet_bootstrap.cc:492] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9: No bootstrap required, opened a new log
I20260620 22:56:36.412379 6244 ts_tablet_manager.cc:1397] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 22:56:36.412834 6243 tablet_bootstrap.cc:492] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: No bootstrap required, opened a new log
I20260620 22:56:36.412897 6243 ts_tablet_manager.cc:1397] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:56:36.414552 6247 tablet_bootstrap.cc:492] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: Bootstrap starting.
I20260620 22:56:36.414872 6244 raft_consensus.cc:348] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.415083 6244 raft_consensus.cc:374] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:36.415131 6244 raft_consensus.cc:729] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 168b65e2251643a980cb537232577be9, State: Initialized, Role: FOLLOWER
I20260620 22:56:36.415216 6247 tablet_bootstrap.cc:654] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:36.415113 6243 raft_consensus.cc:348] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.415259 6243 raft_consensus.cc:374] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:36.415298 6243 raft_consensus.cc:729] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee664cd23d3401b81bbea2e5977ea3f, State: Initialized, Role: FOLLOWER
I20260620 22:56:36.415295 6244 consensus_queue.cc:260] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.415427 6243 consensus_queue.cc:260] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.415717 6243 ts_tablet_manager.cc:1428] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:56:36.415709 5960 heartbeater.cc:503] Master 127.1.166.62:38803 was elected leader, sending a full tablet report...
I20260620 22:56:36.415812 6247 log.cc:824] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:36.415850 6093 heartbeater.cc:503] Master 127.1.166.62:38803 was elected leader, sending a full tablet report...
I20260620 22:56:36.415671 6244 ts_tablet_manager.cc:1428] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 22:56:36.416893 6247 tablet_bootstrap.cc:492] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: No bootstrap required, opened a new log
I20260620 22:56:36.416966 6247 ts_tablet_manager.cc:1397] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:56:36.418841 6247 raft_consensus.cc:348] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.419049 6247 raft_consensus.cc:374] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:36.419085 6247 raft_consensus.cc:729] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16cf613e8a9d4b45ac5a54d17f7bc256, State: Initialized, Role: FOLLOWER
I20260620 22:56:36.419221 6247 consensus_queue.cc:260] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.419572 6247 ts_tablet_manager.cc:1428] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:56:36.419595 6226 heartbeater.cc:503] Master 127.1.166.62:38803 was elected leader, sending a full tablet report...
W20260620 22:56:36.435326 6094 tablet.cc:2369] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:36.496886 6251 raft_consensus.cc:482] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:36.497022 6251 raft_consensus.cc:504] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.497462 6251 leader_election.cc:288] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 168b65e2251643a980cb537232577be9 (127.1.166.1:45967), 1ee664cd23d3401b81bbea2e5977ea3f (127.1.166.2:45885)
I20260620 22:56:36.502182 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.497776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:56657 (local address 127.1.166.2:45885)
0620 22:56:36.497871 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.497873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.497887 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:56:36.497925 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.497926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.497940 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.498012 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:36.498211 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.499107 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.501049 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.501706 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.501790 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.501803 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.502042 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.502093 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 22:56:36.502182 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.497784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:38627 (local address 127.1.166.1:45967)
0620 22:56:36.497840 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.497841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.498034 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:56:36.498074 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.498075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.498090 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.498140 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:36.498290 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.499094 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.501048 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.501691 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.501790 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.501803 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.502042 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.502093 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:56:36.502290 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.497672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45885 (local address 127.1.166.3:56657)
0620 22:56:36.497739 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.497752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.497800 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.498036 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.498037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.498039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:36.498124 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.498126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.499244 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.499245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.500847 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.500861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.501835 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.501837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.501942 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.501946 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.502017 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.502159 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":179}
I20260620 22:56:36.502305 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.497707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:45967 (local address 127.1.166.3:38627)
0620 22:56:36.497947 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.497956 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.498013 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.498160 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.498161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.498162 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:36.498218 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.498220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.499244 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.499245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.500847 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:36.500855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.501835 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.501837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:36.501941 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.501946 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.502017 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.502149 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":3,"thread_start_us":171,"threads_started":1}
I20260620 22:56:36.502903 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd086e44f254e818d5d1ab77b4f9240" candidate_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" is_pre_election: true
I20260620 22:56:36.503109 6047 raft_consensus.cc:2454] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16cf613e8a9d4b45ac5a54d17f7bc256 in term 0.
I20260620 22:56:36.503013 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd086e44f254e818d5d1ab77b4f9240" candidate_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "168b65e2251643a980cb537232577be9" is_pre_election: true
I20260620 22:56:36.503378 5914 raft_consensus.cc:2454] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16cf613e8a9d4b45ac5a54d17f7bc256 in term 0.
I20260620 22:56:36.503446 6115 leader_election.cc:302] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16cf613e8a9d4b45ac5a54d17f7bc256, 1ee664cd23d3401b81bbea2e5977ea3f; no voters:
I20260620 22:56:36.503688 6251 raft_consensus.cc:2781] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:36.503744 6251 raft_consensus.cc:482] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:36.503762 6251 raft_consensus.cc:3037] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:36.504628 6251 raft_consensus.cc:504] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.504781 6251 leader_election.cc:288] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [CANDIDATE]: Term 1 election: Requested vote from peers 168b65e2251643a980cb537232577be9 (127.1.166.1:45967), 1ee664cd23d3401b81bbea2e5977ea3f (127.1.166.2:45885)
I20260620 22:56:36.504959 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd086e44f254e818d5d1ab77b4f9240" candidate_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "168b65e2251643a980cb537232577be9"
I20260620 22:56:36.505045 6047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd086e44f254e818d5d1ab77b4f9240" candidate_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee664cd23d3401b81bbea2e5977ea3f"
I20260620 22:56:36.505059 5914 raft_consensus.cc:3037] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:36.505326 6047 raft_consensus.cc:3037] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:36.506295 6047 raft_consensus.cc:2454] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16cf613e8a9d4b45ac5a54d17f7bc256 in term 1.
I20260620 22:56:36.506295 5914 raft_consensus.cc:2454] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16cf613e8a9d4b45ac5a54d17f7bc256 in term 1.
I20260620 22:56:36.506520 6116 leader_election.cc:302] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 168b65e2251643a980cb537232577be9, 16cf613e8a9d4b45ac5a54d17f7bc256; no voters:
I20260620 22:56:36.506726 6251 raft_consensus.cc:2781] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:36.506991 6251 raft_consensus.cc:686] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 1 LEADER]: Becoming Leader. State: Replica: 16cf613e8a9d4b45ac5a54d17f7bc256, State: Running, Role: LEADER
I20260620 22:56:36.507167 6251 consensus_queue.cc:237] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } }
I20260620 22:56:36.508291 5770 catalog_manager.cc:5495] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16cf613e8a9d4b45ac5a54d17f7bc256 (127.1.166.3). New cstate: current_term: 1 leader_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16cf613e8a9d4b45ac5a54d17f7bc256" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 } health_report { overall_health: UNKNOWN } } }
I20260620 22:56:36.525108 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.521926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.1.166.3:46021)
0620 22:56:36.521994 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.521995 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.522019 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:56:36.522064 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.522065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.522089 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.522150 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:36.522391 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.523032 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.523859 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:36.524084 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:36.524184 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:36.524477 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:36.524670 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:36.524766 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.524885 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.524914 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 22:56:36.525310 5961 tablet.cc:2369] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:36.527422 6047 raft_consensus.cc:1261] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 1 FOLLOWER]: Refusing update from remote peer 16cf613e8a9d4b45ac5a54d17f7bc256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:36.527422 5914 raft_consensus.cc:1261] T 7bd086e44f254e818d5d1ab77b4f9240 P 168b65e2251643a980cb537232577be9 [term 1 FOLLOWER]: Refusing update from remote peer 16cf613e8a9d4b45ac5a54d17f7bc256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:36.527853 6251 consensus_queue.cc:1035] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee664cd23d3401b81bbea2e5977ea3f" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:36.528004 6253 consensus_queue.cc:1035] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "168b65e2251643a980cb537232577be9" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:36.530961 6258 mvcc.cc:205] Tried to move back new op lower bound from 7299056420973326336 to 7299056420894318592. Current Snapshot: MvccSnapshot[applied={T|T < 7299056420973326336}]
I20260620 22:56:36.532939 5770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:36.533051 5770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:36.533833 5770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fca3417b80ea468095ab838da4124ff7: Sending DeleteTablet for 3 replicas of tablet 7bd086e44f254e818d5d1ab77b4f9240
I20260620 22:56:36.534202 5894 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd086e44f254e818d5d1ab77b4f9240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:36 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:50100
I20260620 22:56:36.534271 6160 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd086e44f254e818d5d1ab77b4f9240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:36 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:33448
I20260620 22:56:36.534324 6027 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd086e44f254e818d5d1ab77b4f9240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:36 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:39866
I20260620 22:56:36.534650 6267 tablet_replica.cc:330] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: stopping tablet replica
I20260620 22:56:36.534762 6267 raft_consensus.cc:2229] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:56:36.534899 6267 raft_consensus.cc:2258] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:36.535053 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 5832
I20260620 22:56:36.535771 6267 ts_tablet_manager.cc:1905] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:36.537029 6268 tablet_replica.cc:330] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: stopping tablet replica
I20260620 22:56:36.537249 6268 raft_consensus.cc:2229] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:36.537446 6268 raft_consensus.cc:2258] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:36.537606 6267 ts_tablet_manager.cc:1918] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:36.537683 6267 log.cc:1196] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/7bd086e44f254e818d5d1ab77b4f9240
I20260620 22:56:36.537928 6267 ts_tablet_manager.cc:1939] T 7bd086e44f254e818d5d1ab77b4f9240 P 16cf613e8a9d4b45ac5a54d17f7bc256: Deleting consensus metadata
I20260620 22:56:36.539003 5756 catalog_manager.cc:4841] TS 16cf613e8a9d4b45ac5a54d17f7bc256 (127.1.166.3:46021): tablet 7bd086e44f254e818d5d1ab77b4f9240 (table test-table [id=a9de9847a24e492db60cd150cd989826]) successfully deleted
I20260620 22:56:36.539202 6268 ts_tablet_manager.cc:1905] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:36.540669 6268 ts_tablet_manager.cc:1918] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:36.540750 6268 log.cc:1196] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/7bd086e44f254e818d5d1ab77b4f9240
I20260620 22:56:36.540992 6268 ts_tablet_manager.cc:1939] T 7bd086e44f254e818d5d1ab77b4f9240 P 1ee664cd23d3401b81bbea2e5977ea3f: Deleting consensus metadata
I20260620 22:56:36.541452 5756 catalog_manager.cc:4841] TS 1ee664cd23d3401b81bbea2e5977ea3f (127.1.166.2:45885): tablet 7bd086e44f254e818d5d1ab77b4f9240 (table test-table [id=a9de9847a24e492db60cd150cd989826]) successfully deleted
W20260620 22:56:36.541707 5757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:45967 (error 108)
I20260620 22:56:36.541859 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 5965
I20260620 22:56:36.542675 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.542513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:45967 (local address 127.0.0.1:50108)
0620 22:56:36.542578 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.542630 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:45967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 22:56:36.542893 5757 catalog_manager.cc:4584] TS 168b65e2251643a980cb537232577be9 (127.1.166.1:45967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bd086e44f254e818d5d1ab77b4f9240: Network error: Client connection negotiation failed: client connection to 127.1.166.1:45967: connect: Connection refused (error 111)
I20260620 22:56:36.546576 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 6098
I20260620 22:56:36.551569 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 5739
2026-06-20T22:56:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1962 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T22:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:36Z Disabled control of system clock
I20260620 22:56:36.578485 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35469
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:35469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:56:36.664394 6273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:36.664629 6273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:36.664650 6273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:36.665827 6273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:36.665869 6273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:36.665889 6273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:36.665900 6273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:36.665910 6273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:36.667331 6273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:35469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:36.667570 6273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:36.667833 6273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:36.672482 6279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.672458 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.672458 6278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:36.672995 6273 server_base.cc:1034] running on GCE node
I20260620 22:56:36.673235 6273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:36.673527 6273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:36.674675 6273 hybrid_clock.cc:648] HybridClock initialized: now 1781996196674660 us; error 27 us; skew 500 ppm
I20260620 22:56:36.676323 6273 webserver.cc:460] Webserver started at http://127.1.166.62:40083/ using document root <none> and password file <none>
I20260620 22:56:36.676575 6273 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:36.676626 6273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:36.676756 6273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:36.677848 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "765b0110582037fb3b9d09c91ae2e6e0"
server_key_iv: "559a63528d13bc6cbf62c1c0483bb3dc"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.678186 6273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "765b0110582037fb3b9d09c91ae2e6e0"
server_key_iv: "559a63528d13bc6cbf62c1c0483bb3dc"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.679517 6273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:36.680565 6287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.680791 6273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:36.680882 6273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "765b0110582037fb3b9d09c91ae2e6e0"
server_key_iv: "559a63528d13bc6cbf62c1c0483bb3dc"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.680946 6273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:36.684368 6273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:36.684623 6273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:36.684721 6273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:36.688525 6273 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:35469
I20260620 22:56:36.688602 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:35469 every 8 connection(s)
I20260620 22:56:36.688931 6273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:36.689644 6340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:36.692710 6340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2: Bootstrap starting.
I20260620 22:56:36.693382 6340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:36.693676 6340 log.cc:824] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:36.693698 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 6273
I20260620 22:56:36.693783 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:36.693949 1688 external_mini_cluster.cc:1428] Setting key 5c712b3a720a1dd111b723e330c8ccca
I20260620 22:56:36.694541 6340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2: No bootstrap required, opened a new log
I20260620 22:56:36.696480 6340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35469 } }
I20260620 22:56:36.696676 6340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:36.696700 6340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d890d3f3bf4b3ca9a6f790f726e5b2, State: Initialized, Role: FOLLOWER
I20260620 22:56:36.696838 6340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35469 } }
I20260620 22:56:36.696935 6340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:36.696969 6340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:36.697018 6340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:36.698096 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.694511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.1.166.62:35469)
0620 22:56:36.694725 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.694734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.694760 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:56:36.696474 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.696487 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.696567 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.696783 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:36.697517 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:36.697521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:36.697521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:36.697526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:36.697566 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:36.697578 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:36.697633 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.697715 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.697849 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260620 22:56:36.698169 6340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35469 } }
I20260620 22:56:36.698298 6340 leader_election.cc:302] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d890d3f3bf4b3ca9a6f790f726e5b2; no voters:
I20260620 22:56:36.698505 6340 leader_election.cc:288] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:36.698805 6340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:36.698853 6345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:36.699013 6345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [term 1 LEADER]: Becoming Leader. State: Replica: 83d890d3f3bf4b3ca9a6f790f726e5b2, State: Running, Role: LEADER
I20260620 22:56:36.699110 6345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35469 } }
I20260620 22:56:36.699494 6347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d890d3f3bf4b3ca9a6f790f726e5b2. Latest consensus state: current_term: 1 leader_uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35469 } } }
I20260620 22:56:36.699592 6347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:36.699489 6346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d890d3f3bf4b3ca9a6f790f726e5b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35469 } } }
I20260620 22:56:36.699674 6346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:36.699925 6354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:36.700464 6354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:36.702561 6354 catalog_manager.cc:1347] Generated new cluster ID: 524b7ab30cae4fceb6cd767a6c514cb5
I20260620 22:56:36.702638 6354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:36.720942 6354 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:36.721251 6354 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:36.729115 6354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2: Generated new TSK 0
I20260620 22:56:36.729370 6354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:36.741183 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35469
--builtin_ntp_servers=127.1.166.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:56:36.833251 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:36.833498 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:36.833523 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:36.834774 6364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:36.834813 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:36.834841 6364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:36.836784 6364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:36.837117 6364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:36.837466 6364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:36.842902 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.842914 6370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.842939 6369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:36.843361 6364 server_base.cc:1034] running on GCE node
I20260620 22:56:36.843578 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:36.843816 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:36.844995 6364 hybrid_clock.cc:648] HybridClock initialized: now 1781996196844977 us; error 35 us; skew 500 ppm
I20260620 22:56:36.846522 6364 webserver.cc:460] Webserver started at http://127.1.166.1:38563/ using document root <none> and password file <none>
I20260620 22:56:36.846781 6364 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:36.846837 6364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:36.846961 6364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:36.848057 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "f24cdf2c7f0748b4af0441fc658caace"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "d5a5284ce054fa114dc76b5bfb589f0c"
server_key_iv: "3228f4eb65d5f20cf118225aa8c0c43f"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.848371 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "f24cdf2c7f0748b4af0441fc658caace"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "d5a5284ce054fa114dc76b5bfb589f0c"
server_key_iv: "3228f4eb65d5f20cf118225aa8c0c43f"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.849718 6364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:36.850658 6378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.850848 6364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:36.850942 6364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "f24cdf2c7f0748b4af0441fc658caace"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "d5a5284ce054fa114dc76b5bfb589f0c"
server_key_iv: "3228f4eb65d5f20cf118225aa8c0c43f"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.851009 6364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:36.854655 6364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:36.854972 6364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:36.855104 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:36.855300 6364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:36.855623 6364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:36.855657 6364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.855686 6364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:36.855710 6364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.861367 6364 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:40957
I20260620 22:56:36.861447 6491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:40957 every 8 connection(s)
I20260620 22:56:36.861760 6364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:56:36.864712 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.862534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:38183 (local address 127.1.166.62:35469)
0620 22:56:36.862595 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.862597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.862806 (+ 209us) server_negotiation.cc:407] Connection header received
0620 22:56:36.863308 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.863309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.863317 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.863370 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:36.864494 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:36.864496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:36.864496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:36.864501 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:36.864517 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:36.864521 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:36.864582 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.864640 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.864654 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:56:36.864889 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.862539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35469 (local address 127.1.166.1:38183)
0620 22:56:36.862697 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.862731 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.863197 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.863433 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.863436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.863445 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:36.864331 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:56:36.864331 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:36.864361 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:56:36.864390 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:36.864587 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:36.864590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:36.864592 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.864646 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.864697 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260620 22:56:36.865499 6492 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35469
I20260620 22:56:36.865609 6492 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:36.865765 6492 heartbeater.cc:511] Master 127.1.166.62:35469 requested a full tablet report, sending...
I20260620 22:56:36.866428 6304 ts_manager.cc:194] Registered new tserver with Master: f24cdf2c7f0748b4af0441fc658caace (127.1.166.1:40957)
I20260620 22:56:36.866556 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 6364
I20260620 22:56:36.866657 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:36.866820 1688 external_mini_cluster.cc:1428] Setting key ff8f0266ca7ed03b67ed4171d172b526
I20260620 22:56:36.867761 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35469
--builtin_ntp_servers=127.1.166.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:56:36.951092 6495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:36.951306 6495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:36.951329 6495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:36.952504 6495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:36.952550 6495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:36.952579 6495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:36.954008 6495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:36.954236 6495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:36.954560 6495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:36.959625 6503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:36.959765 6495 server_base.cc:1034] running on GCE node
W20260620 22:56:36.959714 6500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:36.959713 6501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:36.960086 6495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:36.960417 6495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:36.961655 6495 hybrid_clock.cc:648] HybridClock initialized: now 1781996196961598 us; error 74 us; skew 500 ppm
I20260620 22:56:36.962992 6495 webserver.cc:460] Webserver started at http://127.1.166.2:35725/ using document root <none> and password file <none>
I20260620 22:56:36.963220 6495 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:36.963267 6495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:36.963373 6495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:36.964393 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "f12452e664fe4388a849195c3bf3cb1c"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "73f865d39d2ea964c7ab30046c53c555"
server_key_iv: "25bcda94851e63792e716713034f3834"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.964723 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "f12452e664fe4388a849195c3bf3cb1c"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "73f865d39d2ea964c7ab30046c53c555"
server_key_iv: "25bcda94851e63792e716713034f3834"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.965981 6495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:36.966785 6509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.966914 6495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.966989 6495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "f12452e664fe4388a849195c3bf3cb1c"
format_stamp: "Formatted at 2026-06-20 22:56:36 on dist-test-slave-rdl9"
server_key: "73f865d39d2ea964c7ab30046c53c555"
server_key_iv: "25bcda94851e63792e716713034f3834"
server_key_version: "encryptionkey@0"
I20260620 22:56:36.967053 6495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:36.970400 6495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:36.970638 6495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:36.970733 6495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:36.971045 6495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:36.971335 6495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:36.971374 6495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.971400 6495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:36.971416 6495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:36.977125 6495 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:46709
I20260620 22:56:36.977172 6622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:46709 every 8 connection(s)
I20260620 22:56:36.977507 6495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:36.980507 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.978264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:55895 (local address 127.1.166.62:35469)
0620 22:56:36.978361 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:36.978363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:36.978588 (+ 225us) server_negotiation.cc:407] Connection header received
0620 22:56:36.979036 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:36.979037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:36.979042 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:36.979086 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:36.980310 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:36.980312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:36.980312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:36.980317 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:36.980334 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:36.980339 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:36.980388 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:36.980435 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 22:56:36.980447 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 22:56:36.980710 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:36.978213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35469 (local address 127.1.166.2:55895)
0620 22:56:36.978429 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:36.978467 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:36.978940 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:36.979156 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:36.979159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:36.979169 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:36.980133 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:56:36.980133 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:36.980184 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:56:36.980216 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:36.980387 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:36.980390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:36.980392 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:56:36.980444 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:56:36.980496 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260620 22:56:36.981178 6623 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35469
I20260620 22:56:36.981269 6623 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:36.981428 6623 heartbeater.cc:511] Master 127.1.166.62:35469 requested a full tablet report, sending...
I20260620 22:56:36.981813 6304 ts_manager.cc:194] Registered new tserver with Master: f12452e664fe4388a849195c3bf3cb1c (127.1.166.2:46709)
I20260620 22:56:36.982674 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 6495
I20260620 22:56:36.982776 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:36.982945 1688 external_mini_cluster.cc:1428] Setting key 59d24ff9b704834eed811a2e4679ef7f
I20260620 22:56:36.983982 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35469
--builtin_ntp_servers=127.1.166.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:56:37.078708 6626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:37.078965 6626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:37.079003 6626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:37.080420 6626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:37.080479 6626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:37.080521 6626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:37.082113 6626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:37.082456 6626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:37.082785 6626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:37.088438 6632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:37.088399 6634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:37.088382 6626 server_base.cc:1034] running on GCE node
W20260620 22:56:37.088376 6631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:37.089213 6626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:37.089668 6626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:37.090905 6626 hybrid_clock.cc:648] HybridClock initialized: now 1781996197090861 us; error 75 us; skew 500 ppm
I20260620 22:56:37.095248 6626 webserver.cc:460] Webserver started at http://127.1.166.3:42231/ using document root <none> and password file <none>
I20260620 22:56:37.095590 6626 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:37.095652 6626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:37.095793 6626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:37.097018 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "2a6a302dfb444254af8d67fbecaeb85c"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "e2e29e12ab5d4c46ef8dda089d9a7e79"
server_key_iv: "761df2183c8e7a0cd5bc2eb68875effa"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.097385 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "2a6a302dfb444254af8d67fbecaeb85c"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "e2e29e12ab5d4c46ef8dda089d9a7e79"
server_key_iv: "761df2183c8e7a0cd5bc2eb68875effa"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.099010 6626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:37.100018 6640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:37.100211 6626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:37.100317 6626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "2a6a302dfb444254af8d67fbecaeb85c"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "e2e29e12ab5d4c46ef8dda089d9a7e79"
server_key_iv: "761df2183c8e7a0cd5bc2eb68875effa"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.100404 6626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:37.104043 6626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:37.104355 6626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:37.104493 6626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:37.104694 6626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:37.105029 6626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:37.105062 6626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:37.105091 6626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:37.105115 6626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:37.112754 6626 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:36409
I20260620 22:56:37.112833 6753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:36409 every 8 connection(s)
I20260620 22:56:37.113161 6626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:37.117033 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.114051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:49895 (local address 127.1.166.62:35469)
0620 22:56:37.114145 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.114148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.114402 (+ 254us) server_negotiation.cc:407] Connection header received
0620 22:56:37.115097 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.115099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.115107 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.115166 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.116749 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.116751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.116751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.116759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.116779 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.116784 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.116841 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.116929 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.116947 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 22:56:37.117264 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.113941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35469 (local address 127.1.166.3:49895)
0620 22:56:37.114216 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.114272 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.114957 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.115266 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.115272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.115285 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:37.116492 (+ 1207us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:56:37.116494 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:37.116535 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:56:37.116588 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:37.116847 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:37.116853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:37.116856 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:56:37.116945 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 22:56:37.117030 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260620 22:56:37.117856 6754 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35469
I20260620 22:56:37.117951 6754 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:37.118114 6754 heartbeater.cc:511] Master 127.1.166.62:35469 requested a full tablet report, sending...
I20260620 22:56:37.118575 6304 ts_manager.cc:194] Registered new tserver with Master: 2a6a302dfb444254af8d67fbecaeb85c (127.1.166.3:36409)
I20260620 22:56:37.119856 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 6626
I20260620 22:56:37.119942 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:37.120092 1688 external_mini_cluster.cc:1428] Setting key c8c8b4388177666cc5a7f022b7b05453
I20260620 22:56:37.121102 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:56:37.123579 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.122473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.1.166.62:35469)
0620 22:56:37.122543 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.122545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.122642 (+ 97us) server_negotiation.cc:407] Connection header received
0620 22:56:37.122714 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.122715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.122721 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.122767 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.123362 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.123363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.123363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.123368 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.123391 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.123395 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.123441 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.123487 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.123503 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:56:37.125459 6304 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:37.127636 6304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:37.132939 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.130795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:36409 (local address 127.0.0.1:54834)
0620 22:56:37.131043 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.131097 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.131195 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.131767 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.131776 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.131785 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:37.132447 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:56:37.132448 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:37.132473 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:56:37.132504 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:37.132757 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:37.132764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:37.132768 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:37.132834 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 22:56:37.132851 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":153,"threads_started":1}
I20260620 22:56:37.133044 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.131393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.1.166.2:46709)
0620 22:56:37.131699 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.131706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.131720 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:56:37.131750 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.131754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.131766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.131860 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.132694 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.132701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.132701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.132706 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.132742 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.132765 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.132835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.132918 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.132936 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":92,"threads_started":1}
I20260620 22:56:37.133476 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.130941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.1.166.3:36409)
0620 22:56:37.131322 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.131329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.131343 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:56:37.131377 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.131380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.131393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.131487 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.132617 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.132625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.132626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.132631 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.132677 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.132703 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.132765 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.133259 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.133351 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":124,"threads_started":1}
I20260620 22:56:37.132951 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.131125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:46709 (local address 127.0.0.1:42602)
0620 22:56:37.131548 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.131558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.131603 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.131863 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.131864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.131868 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:37.132544 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:56:37.132545 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:37.132564 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:56:37.132593 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:37.132843 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:37.132844 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:37.132845 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:56:37.132895 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:56:37.132905 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":74,"threads_started":1}
I20260620 22:56:37.134049 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.130795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:40957 (local address 127.0.0.1:48252)
0620 22:56:37.131301 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.131310 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.131345 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.133098 (+ 1753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.133100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.133106 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:37.133654 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:56:37.133654 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:37.133663 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:56:37.133675 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:37.133913 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:37.133914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:37.133915 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:56:37.133958 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:56:37.133973 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":2,"thread_start_us":154,"threads_started":1}
I20260620 22:56:37.134320 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.131162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.1.166.1:40957)
0620 22:56:37.132682 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.132687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.132697 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:37.132722 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.132724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.132734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.132810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.133750 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.133758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.133758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.133762 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.133793 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.133822 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.133877 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.134209 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.134225 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1477,"thread_start_us":100,"threads_started":1}
I20260620 22:56:37.134536 6688 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8d2d653b99482f8030f0fd4d3b21e6 (DEFAULT_TABLE table=test-table [id=1f495e06d4ff4487bba3963dc46fbc91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:37.134656 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8d2d653b99482f8030f0fd4d3b21e6 (DEFAULT_TABLE table=test-table [id=1f495e06d4ff4487bba3963dc46fbc91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:37.134845 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8d2d653b99482f8030f0fd4d3b21e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:37.134997 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8d2d653b99482f8030f0fd4d3b21e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:37.137516 6768 tablet_bootstrap.cc:492] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: Bootstrap starting.
I20260620 22:56:37.138063 6769 tablet_bootstrap.cc:492] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c: Bootstrap starting.
I20260620 22:56:37.138504 6768 tablet_bootstrap.cc:654] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:37.138450 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8d2d653b99482f8030f0fd4d3b21e6 (DEFAULT_TABLE table=test-table [id=1f495e06d4ff4487bba3963dc46fbc91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:37.138733 6769 tablet_bootstrap.cc:654] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:37.138857 6768 log.cc:824] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:37.138876 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8d2d653b99482f8030f0fd4d3b21e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:37.139022 6769 log.cc:824] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:37.139775 6769 tablet_bootstrap.cc:492] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c: No bootstrap required, opened a new log
I20260620 22:56:37.139775 6768 tablet_bootstrap.cc:492] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: No bootstrap required, opened a new log
I20260620 22:56:37.139853 6769 ts_tablet_manager.cc:1397] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:56:37.139853 6768 ts_tablet_manager.cc:1397] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:56:37.141906 6772 tablet_bootstrap.cc:492] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace: Bootstrap starting.
I20260620 22:56:37.142642 6772 tablet_bootstrap.cc:654] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:37.142514 6768 raft_consensus.cc:348] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.142517 6769 raft_consensus.cc:348] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.142688 6768 raft_consensus.cc:374] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:37.142688 6769 raft_consensus.cc:374] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:37.142722 6769 raft_consensus.cc:729] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12452e664fe4388a849195c3bf3cb1c, State: Initialized, Role: FOLLOWER
I20260620 22:56:37.142722 6768 raft_consensus.cc:729] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6a302dfb444254af8d67fbecaeb85c, State: Initialized, Role: FOLLOWER
I20260620 22:56:37.142871 6769 consensus_queue.cc:260] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.142871 6768 consensus_queue.cc:260] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.142961 6772 log.cc:824] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:37.143159 6769 ts_tablet_manager.cc:1428] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 22:56:37.143159 6768 ts_tablet_manager.cc:1428] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260620 22:56:37.143280 6623 heartbeater.cc:503] Master 127.1.166.62:35469 was elected leader, sending a full tablet report...
I20260620 22:56:37.143330 6754 heartbeater.cc:503] Master 127.1.166.62:35469 was elected leader, sending a full tablet report...
I20260620 22:56:37.143748 6772 tablet_bootstrap.cc:492] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace: No bootstrap required, opened a new log
I20260620 22:56:37.143831 6772 ts_tablet_manager.cc:1397] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:56:37.145887 6772 raft_consensus.cc:348] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.146018 6772 raft_consensus.cc:374] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:37.146039 6772 raft_consensus.cc:729] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24cdf2c7f0748b4af0441fc658caace, State: Initialized, Role: FOLLOWER
I20260620 22:56:37.146136 6772 consensus_queue.cc:260] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.146380 6772 ts_tablet_manager.cc:1428] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 22:56:37.146490 6492 heartbeater.cc:503] Master 127.1.166.62:35469 was elected leader, sending a full tablet report...
I20260620 22:56:37.179325 6774 raft_consensus.cc:482] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:37.179491 6774 raft_consensus.cc:504] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.179872 6774 leader_election.cc:288] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a6a302dfb444254af8d67fbecaeb85c (127.1.166.3:36409), f24cdf2c7f0748b4af0441fc658caace (127.1.166.1:40957)
I20260620 22:56:37.181339 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.180129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:36409 (local address 127.1.166.2:56183)
0620 22:56:37.180198 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.180210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.180251 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.180424 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.180426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.180429 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:37.180956 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:56:37.180957 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:37.180965 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:56:37.180978 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:37.181210 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:37.181211 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:37.181212 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:56:37.181267 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:56:37.181280 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 22:56:37.181339 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.180207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:56183 (local address 127.1.166.3:36409)
0620 22:56:37.180268 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.180269 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.180309 (+ 40us) server_negotiation.cc:407] Connection header received
0620 22:56:37.180336 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.180336 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.180341 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.180394 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.181092 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.181093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.181094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.181098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.181117 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.181121 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.181183 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.181274 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.181285 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:56:37.181568 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.180189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:38995 (local address 127.1.166.1:40957)
0620 22:56:37.180259 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.180261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.180532 (+ 271us) server_negotiation.cc:407] Connection header received
0620 22:56:37.180565 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.180567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.180572 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.180625 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.181102 (+ 477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.181103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.181103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.181107 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.181121 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.181125 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.181179 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.181486 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.181502 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:56:37.181583 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.180131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:40957 (local address 127.1.166.2:38995)
0620 22:56:37.180458 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.180468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.180520 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.180654 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.180655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.180658 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:37.181027 (+ 369us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:56:37.181027 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:37.181033 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:56:37.181044 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:37.181428 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:37.181429 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:37.181429 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 22:56:37.181478 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:56:37.181488 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":161,"thread_start_us":103,"threads_started":1}
I20260620 22:56:37.181767 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8d2d653b99482f8030f0fd4d3b21e6" candidate_uuid: "f12452e664fe4388a849195c3bf3cb1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6a302dfb444254af8d67fbecaeb85c" is_pre_election: true
I20260620 22:56:37.181874 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8d2d653b99482f8030f0fd4d3b21e6" candidate_uuid: "f12452e664fe4388a849195c3bf3cb1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24cdf2c7f0748b4af0441fc658caace" is_pre_election: true
I20260620 22:56:37.181963 6708 raft_consensus.cc:2454] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12452e664fe4388a849195c3bf3cb1c in term 0.
I20260620 22:56:37.182032 6446 raft_consensus.cc:2454] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12452e664fe4388a849195c3bf3cb1c in term 0.
I20260620 22:56:37.182235 6513 leader_election.cc:302] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6a302dfb444254af8d67fbecaeb85c, f12452e664fe4388a849195c3bf3cb1c; no voters:
I20260620 22:56:37.182402 6774 raft_consensus.cc:2781] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:37.182471 6774 raft_consensus.cc:482] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:37.182487 6774 raft_consensus.cc:3037] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:37.183223 6774 raft_consensus.cc:504] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.183353 6774 leader_election.cc:288] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [CANDIDATE]: Term 1 election: Requested vote from peers 2a6a302dfb444254af8d67fbecaeb85c (127.1.166.3:36409), f24cdf2c7f0748b4af0441fc658caace (127.1.166.1:40957)
I20260620 22:56:37.183535 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8d2d653b99482f8030f0fd4d3b21e6" candidate_uuid: "f12452e664fe4388a849195c3bf3cb1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6a302dfb444254af8d67fbecaeb85c"
I20260620 22:56:37.183535 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8d2d653b99482f8030f0fd4d3b21e6" candidate_uuid: "f12452e664fe4388a849195c3bf3cb1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24cdf2c7f0748b4af0441fc658caace"
I20260620 22:56:37.183632 6708 raft_consensus.cc:3037] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:37.183642 6446 raft_consensus.cc:3037] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:37.184569 6708 raft_consensus.cc:2454] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12452e664fe4388a849195c3bf3cb1c in term 1.
I20260620 22:56:37.184569 6446 raft_consensus.cc:2454] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12452e664fe4388a849195c3bf3cb1c in term 1.
I20260620 22:56:37.184810 6513 leader_election.cc:302] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6a302dfb444254af8d67fbecaeb85c, f12452e664fe4388a849195c3bf3cb1c; no voters:
I20260620 22:56:37.184921 6774 raft_consensus.cc:2781] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:37.185086 6774 raft_consensus.cc:686] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 1 LEADER]: Becoming Leader. State: Replica: f12452e664fe4388a849195c3bf3cb1c, State: Running, Role: LEADER
I20260620 22:56:37.185194 6774 consensus_queue.cc:237] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } }
I20260620 22:56:37.186096 6302 catalog_manager.cc:5495] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c reported cstate change: term changed from 0 to 1, leader changed from <none> to f12452e664fe4388a849195c3bf3cb1c (127.1.166.2). New cstate: current_term: 1 leader_uuid: "f12452e664fe4388a849195c3bf3cb1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12452e664fe4388a849195c3bf3cb1c" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46709 } health_report { overall_health: HEALTHY } } }
I20260620 22:56:37.198436 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.197369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.1.166.2:46709)
0620 22:56:37.197422 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.197424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.197432 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:37.197470 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.197471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.197485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.197557 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.198150 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.198151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.198151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.198155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.198170 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.198174 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.198220 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.198319 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.198330 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:56:37.200874 6708 raft_consensus.cc:1261] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 1 FOLLOWER]: Refusing update from remote peer f12452e664fe4388a849195c3bf3cb1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:37.200874 6446 raft_consensus.cc:1261] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f24cdf2c7f0748b4af0441fc658caace [term 1 FOLLOWER]: Refusing update from remote peer f12452e664fe4388a849195c3bf3cb1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:37.201267 6774 consensus_queue.cc:1035] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24cdf2c7f0748b4af0441fc658caace" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 40957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:37.201424 6778 consensus_queue.cc:1035] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6a302dfb444254af8d67fbecaeb85c" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:37.204635 6781 mvcc.cc:205] Tried to move back new op lower bound from 7299056423731032064 to 7299056423671103488. Current Snapshot: MvccSnapshot[applied={T|T < 7299056423731032064}]
I20260620 22:56:37.206523 6782 mvcc.cc:205] Tried to move back new op lower bound from 7299056423731032064 to 7299056423671103488. Current Snapshot: MvccSnapshot[applied={T|T < 7299056423731032064 or (T in {7299056423731032064})}]
I20260620 22:56:37.209723 6302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:37.209877 6302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:37.210835 6302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83d890d3f3bf4b3ca9a6f790f726e5b2: Sending DeleteTablet for 3 replicas of tablet 2c8d2d653b99482f8030f0fd4d3b21e6
I20260620 22:56:37.211366 6688 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8d2d653b99482f8030f0fd4d3b21e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:37 UTC) from {username='slave'} at 127.0.0.1:54834
I20260620 22:56:37.211344 6426 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8d2d653b99482f8030f0fd4d3b21e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:37 UTC) from {username='slave'} at 127.0.0.1:48252
I20260620 22:56:37.211848 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 6364
I20260620 22:56:37.211998 6791 tablet_replica.cc:330] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: stopping tablet replica
I20260620 22:56:37.211990 6557 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8d2d653b99482f8030f0fd4d3b21e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:37 UTC) from {username='slave'} at 127.0.0.1:42602
I20260620 22:56:37.212126 6791 raft_consensus.cc:2229] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:37.212252 6791 raft_consensus.cc:2258] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:37.212797 6791 ts_tablet_manager.cc:1905] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:37.214552 6791 ts_tablet_manager.cc:1918] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:37.214650 6791 log.cc:1196] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/2c8d2d653b99482f8030f0fd4d3b21e6
I20260620 22:56:37.214942 6791 ts_tablet_manager.cc:1939] T 2c8d2d653b99482f8030f0fd4d3b21e6 P 2a6a302dfb444254af8d67fbecaeb85c: Deleting consensus metadata
I20260620 22:56:37.215505 6291 catalog_manager.cc:4841] TS 2a6a302dfb444254af8d67fbecaeb85c (127.1.166.3:36409): tablet 2c8d2d653b99482f8030f0fd4d3b21e6 (table test-table [id=1f495e06d4ff4487bba3963dc46fbc91]) successfully deleted
I20260620 22:56:37.218932 6793 tablet_replica.cc:330] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c: stopping tablet replica
W20260620 22:56:37.219034 6290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:40957 (error 108)
I20260620 22:56:37.219094 6793 raft_consensus.cc:2229] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:56:37.219283 6793 raft_consensus.cc:2258] T 2c8d2d653b99482f8030f0fd4d3b21e6 P f12452e664fe4388a849195c3bf3cb1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:37.219364 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 6495
I20260620 22:56:37.220181 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.220000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:40957 (local address 127.0.0.1:48266)
0620 22:56:37.220080 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.220140 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 22:56:37.220314 6290 catalog_manager.cc:4584] TS f24cdf2c7f0748b4af0441fc658caace (127.1.166.1:40957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c8d2d653b99482f8030f0fd4d3b21e6: Network error: Client connection negotiation failed: client connection to 127.1.166.1:40957: connect: Connection refused (error 111)
I20260620 22:56:37.224614 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 6626
I20260620 22:56:37.224728 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.224532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:46709 (local address 127.0.0.1:42614)
0620 22:56:37.224634 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.224680 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.2:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 22:56:37.231024 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 6273
2026-06-20T22:56:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (679 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 22:56:37.251681 1688 mini_oidc.cc:152] Starting JWKS server
I20260620 22:56:37.252926 1688 webserver.cc:460] Webserver started at https://127.0.0.1:46775/ using document root <none> and password file <none>
I20260620 22:56:37.253028 1688 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:56:37.253204 1688 webserver.cc:460] Webserver started at http://127.0.0.1:36677/ using document root <none> and password file <none>
2026-06-20T22:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:37Z Disabled control of system clock
I20260620 22:56:37.263343 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46375
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:46375
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46775/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/testchainca.cert with env {}
W20260620 22:56:37.354080 6800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:37.354324 6800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:37.354377 6800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:37.355690 6800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:37.355736 6800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:37.355747 6800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:37.355759 6800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:37.355770 6800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:56:37.355782 6800 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:56:37.355791 6800 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46775/jwks.json
I20260620 22:56:37.357465 6800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:46375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46775/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:37.357715 6800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:37.357950 6800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:37.362602 6806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:37.362604 6805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:37.362882 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:37.362895 6800 server_base.cc:1034] running on GCE node
I20260620 22:56:37.363188 6800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:37.363509 6800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:37.364651 6800 hybrid_clock.cc:648] HybridClock initialized: now 1781996197364635 us; error 27 us; skew 500 ppm
I20260620 22:56:37.365845 6800 webserver.cc:460] Webserver started at http://127.1.166.62:37455/ using document root <none> and password file <none>
I20260620 22:56:37.366057 6800 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:37.366106 6800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:37.366228 6800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:37.367357 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "e9f8dda311cc426b937f6faaf227441f"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "30c02c5496effc3b8cdac6e25c128b05"
server_key_iv: "a5259ee33d79962fbd11690323f49b10"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.367693 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "e9f8dda311cc426b937f6faaf227441f"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "30c02c5496effc3b8cdac6e25c128b05"
server_key_iv: "a5259ee33d79962fbd11690323f49b10"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.368969 6800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:56:37.369812 6814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:37.370002 6800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:37.370091 6800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "e9f8dda311cc426b937f6faaf227441f"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "30c02c5496effc3b8cdac6e25c128b05"
server_key_iv: "a5259ee33d79962fbd11690323f49b10"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.370160 6800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:37.389282 6800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:37.389812 6800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:37.389976 6800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:37.394627 6800 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:46375
I20260620 22:56:37.394703 6866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:46375 every 8 connection(s)
I20260620 22:56:37.395056 6800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:37.396080 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:37.398454 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f: Bootstrap starting.
I20260620 22:56:37.399030 6867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:37.399304 6867 log.cc:824] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:37.400019 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 6800
I20260620 22:56:37.400079 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f: No bootstrap required, opened a new log
I20260620 22:56:37.400117 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:37.400302 1688 external_mini_cluster.cc:1428] Setting key 1aea067ebcc5d611a6f0ecc87638a12f
I20260620 22:56:37.401885 6867 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8dda311cc426b937f6faaf227441f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46375 } }
I20260620 22:56:37.402017 6867 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:37.402043 6867 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8dda311cc426b937f6faaf227441f, State: Initialized, Role: FOLLOWER
I20260620 22:56:37.402161 6867 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8dda311cc426b937f6faaf227441f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46375 } }
I20260620 22:56:37.402237 6867 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:37.402273 6867 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:37.402314 6867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:37.402858 6867 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8dda311cc426b937f6faaf227441f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46375 } }
I20260620 22:56:37.402944 6867 leader_election.cc:302] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f8dda311cc426b937f6faaf227441f; no voters:
I20260620 22:56:37.403105 6867 leader_election.cc:288] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:37.403143 6872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:37.403254 6872 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [term 1 LEADER]: Becoming Leader. State: Replica: e9f8dda311cc426b937f6faaf227441f, State: Running, Role: LEADER
I20260620 22:56:37.403332 6872 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8dda311cc426b937f6faaf227441f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46375 } }
I20260620 22:56:37.403384 6867 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:37.403625 6872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f8dda311cc426b937f6faaf227441f. Latest consensus state: current_term: 1 leader_uuid: "e9f8dda311cc426b937f6faaf227441f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8dda311cc426b937f6faaf227441f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46375 } } }
I20260620 22:56:37.403697 6872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:37.403635 6874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f8dda311cc426b937f6faaf227441f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f8dda311cc426b937f6faaf227441f" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46375 } } }
I20260620 22:56:37.403807 6874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:37.404011 6876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:37.404549 6876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:37.406178 6876 catalog_manager.cc:1347] Generated new cluster ID: 1aa2fb754ba0477c9216303bcd75b8b8
I20260620 22:56:37.406237 6876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:37.407636 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.400807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.1.166.62:46375)
0620 22:56:37.400991 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.400999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.401023 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:56:37.401879 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.401886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.401948 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.402083 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.402302 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.404026 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.405140 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.405669 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.405783 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.406906 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.406910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.406910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.407081 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.407114 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.407125 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.407182 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.407346 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.407432 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260620 22:56:37.408488 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:37.423168 6876 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:37.423748 6876 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:37.426900 6876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f8dda311cc426b937f6faaf227441f: Generated new TSK 0
I20260620 22:56:37.427124 6876 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 22:56:37.459730 6897 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 22:56:37.459834 6897 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 22:56:37.460454 6896 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:37.443482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46375 (local address 127.0.0.1:51558)
0620 22:56:37.443675 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.443686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.443762 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.443995 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.443997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.444005 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:56:37.444073 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:37.444075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.445063 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.445065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:37.445721 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:37.445731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.446281 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.446283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:37.446417 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.446467 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:56:37.459994 (+ 13527us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 22:56:37.460103 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.166.62:46375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260620 22:56:37.460767 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.443601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.1.166.62:46375)
0620 22:56:37.443655 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.443657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.443748 (+ 91us) server_negotiation.cc:407] Connection header received
0620 22:56:37.443853 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.443854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.443866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.443955 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:56:37.444149 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.444955 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.445824 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.446150 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.446222 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.446695 (+ 473us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:56:37.459832 (+ 13137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 22:56:37.460695 (+ 863us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51558: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 22:56:37.460813 6871 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51558: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.166.62:46375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46775/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 22:56:37.461119 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 6800
2026-06-20T22:56:37Z chronyd exiting
I20260620 22:56:37.496726 1688 test_util.cc:175] -----------------------------------------------
I20260620 22:56:37.496810 1688 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996175763767-1688-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (246 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 22:56:37.497577 1688 mini_oidc.cc:152] Starting JWKS server
I20260620 22:56:37.498266 1688 webserver.cc:460] Webserver started at https://127.0.0.1:37817/ using document root <none> and password file <none>
I20260620 22:56:37.498309 1688 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:56:37.498483 1688 webserver.cc:460] Webserver started at http://127.0.0.1:43441/ using document root <none> and password file <none>
2026-06-20T22:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:37Z Disabled control of system clock
I20260620 22:56:37.506495 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45847
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:45847
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37817/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/testchainca.cert with env {}
W20260620 22:56:37.596444 6903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:37.596730 6903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:37.596760 6903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:37.598671 6903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:37.598740 6903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:37.598753 6903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:37.598766 6903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:37.598778 6903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:56:37.598790 6903 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:56:37.598800 6903 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37817/jwks.json
I20260620 22:56:37.600680 6903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:45847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37817/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:37.600950 6903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:37.601140 6903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:37.606271 6911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:37.606277 6908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:37.606276 6909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:37.606549 6903 server_base.cc:1034] running on GCE node
I20260620 22:56:37.606773 6903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:37.607077 6903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:37.608242 6903 hybrid_clock.cc:648] HybridClock initialized: now 1781996197608211 us; error 48 us; skew 500 ppm
I20260620 22:56:37.609587 6903 webserver.cc:460] Webserver started at http://127.1.166.62:46031/ using document root <none> and password file <none>
I20260620 22:56:37.609833 6903 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:37.609885 6903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:37.610023 6903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:37.611194 6903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "aee14f0fc9574db6b239cc4154d750be"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "deef2beb4e7ced8e3c6421719c72e46f"
server_key_iv: "a15beec7dc41f5ad99cf9dc72629d7d2"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.611567 6903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "aee14f0fc9574db6b239cc4154d750be"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "deef2beb4e7ced8e3c6421719c72e46f"
server_key_iv: "a15beec7dc41f5ad99cf9dc72629d7d2"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.612896 6903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:56:37.613644 6917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:37.613816 6903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:37.613899 6903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "aee14f0fc9574db6b239cc4154d750be"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "deef2beb4e7ced8e3c6421719c72e46f"
server_key_iv: "a15beec7dc41f5ad99cf9dc72629d7d2"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.613963 6903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:37.629140 6903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:37.629514 6903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:37.629657 6903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:37.634169 6903 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:45847
I20260620 22:56:37.634212 6969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:45847 every 8 connection(s)
I20260620 22:56:37.634639 6903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:37.635658 6970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:37.638054 6970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be: Bootstrap starting.
I20260620 22:56:37.638703 6970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:37.638988 6970 log.cc:824] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:37.639765 6970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be: No bootstrap required, opened a new log
I20260620 22:56:37.641510 6970 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee14f0fc9574db6b239cc4154d750be" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45847 } }
I20260620 22:56:37.641732 6970 raft_consensus.cc:374] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:37.641757 6970 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee14f0fc9574db6b239cc4154d750be, State: Initialized, Role: FOLLOWER
I20260620 22:56:37.641922 6970 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee14f0fc9574db6b239cc4154d750be" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45847 } }
I20260620 22:56:37.641997 6970 raft_consensus.cc:388] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:37.642027 6970 raft_consensus.cc:482] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:37.642076 6970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:37.642092 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 6903
I20260620 22:56:37.642169 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:37.642380 1688 external_mini_cluster.cc:1428] Setting key f4c501c16456c7a4164e0b5bb658ce45
I20260620 22:56:37.642848 6970 raft_consensus.cc:504] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee14f0fc9574db6b239cc4154d750be" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45847 } }
I20260620 22:56:37.642984 6970 leader_election.cc:302] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aee14f0fc9574db6b239cc4154d750be; no voters:
I20260620 22:56:37.643157 6970 leader_election.cc:288] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:37.643199 6975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:37.643406 6975 raft_consensus.cc:686] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [term 1 LEADER]: Becoming Leader. State: Replica: aee14f0fc9574db6b239cc4154d750be, State: Running, Role: LEADER
I20260620 22:56:37.643520 6970 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:37.643535 6975 consensus_queue.cc:237] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee14f0fc9574db6b239cc4154d750be" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45847 } }
I20260620 22:56:37.644098 6977 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [sys.catalog]: SysCatalogTable state changed. Reason: New leader aee14f0fc9574db6b239cc4154d750be. Latest consensus state: current_term: 1 leader_uuid: "aee14f0fc9574db6b239cc4154d750be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee14f0fc9574db6b239cc4154d750be" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45847 } } }
I20260620 22:56:37.644275 6977 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:37.644085 6976 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aee14f0fc9574db6b239cc4154d750be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee14f0fc9574db6b239cc4154d750be" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45847 } } }
I20260620 22:56:37.644454 6976 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:37.644692 6984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:37.645298 6984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:37.646724 6984 catalog_manager.cc:1347] Generated new cluster ID: 6e3e9fca7d9147f3ba5ae949e837117b
I20260620 22:56:37.646770 6984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:37.649168 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.642869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.1.166.62:45847)
0620 22:56:37.643097 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.643106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.643139 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:56:37.643262 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.643269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.643340 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.643740 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.643834 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.645517 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.646984 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.647579 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.647691 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.648352 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.648355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.648355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.648556 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.648593 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.648607 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.648832 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.648862 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.648943 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260620 22:56:37.650089 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:37.658210 6984 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:37.658767 6984 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:37.661717 6984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aee14f0fc9574db6b239cc4154d750be: Generated new TSK 0
I20260620 22:56:37.661907 6984 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 22:56:37.686997 7000 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 22:56:37.687291 7000 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 22:56:37.687738 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.672821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.1.166.62:45847)
0620 22:56:37.672873 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.672876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.673019 (+ 143us) server_negotiation.cc:407] Connection header received
0620 22:56:37.673070 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.673071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.673089 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.673165 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:56:37.673309 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.673821 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.675045 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.675266 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.675327 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.675567 (+ 240us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:56:37.687456 (+ 11889us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 22:56:37.687669 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 22:56:37.687796 6974 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 22:56:37.687920 6999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:37.672704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45847 (local address 127.0.0.1:35908)
0620 22:56:37.672893 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.672907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.673001 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.673173 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.673173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.673175 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:56:37.673241 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:37.673244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.673899 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.673900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:37.674912 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:37.674916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.675341 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.675342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:37.675410 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.675417 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:56:37.687624 (+ 12207us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 22:56:37.687799 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.166.62:45847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37817/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260620 22:56:37.688421 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 6903
2026-06-20T22:56:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 22:56:37.726679 1688 mini_oidc.cc:152] Starting JWKS server
I20260620 22:56:37.727360 1688 webserver.cc:460] Webserver started at https://127.0.0.1:37487/ using document root <none> and password file <none>
I20260620 22:56:37.727402 1688 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:56:37.727515 1688 webserver.cc:460] Webserver started at http://127.0.0.1:39367/ using document root <none> and password file <none>
2026-06-20T22:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:37Z Disabled control of system clock
I20260620 22:56:37.736786 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:40255
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:40255
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37487/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 22:56:37.827929 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:37.828258 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:37.828299 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:37.831008 7006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:37.831105 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:37.831125 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:37.831146 7006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:37.831162 7006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:56:37.831180 7006 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:56:37.831195 7006 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37487/jwks.json
I20260620 22:56:37.833410 7006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:40255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37487/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:37.834259 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:37.834844 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:37.839754 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:37.839787 7006 server_base.cc:1034] running on GCE node
W20260620 22:56:37.839753 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:37.839753 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:37.840242 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:37.840555 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:37.841720 7006 hybrid_clock.cc:648] HybridClock initialized: now 1781996197841691 us; error 48 us; skew 500 ppm
I20260620 22:56:37.843036 7006 webserver.cc:460] Webserver started at http://127.1.166.62:39481/ using document root <none> and password file <none>
I20260620 22:56:37.843276 7006 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:37.843327 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:37.843439 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:37.844683 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "cd512a64df894032b434c0fa2e9f5ac3"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "9f29a8943b59b69ee7b7c5e8f3703ea2"
server_key_iv: "d4cf36ac2de815919ddf1eb5df3c63eb"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.845043 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "cd512a64df894032b434c0fa2e9f5ac3"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "9f29a8943b59b69ee7b7c5e8f3703ea2"
server_key_iv: "d4cf36ac2de815919ddf1eb5df3c63eb"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.846405 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:37.847208 7020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:37.847455 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:37.847540 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "cd512a64df894032b434c0fa2e9f5ac3"
format_stamp: "Formatted at 2026-06-20 22:56:37 on dist-test-slave-rdl9"
server_key: "9f29a8943b59b69ee7b7c5e8f3703ea2"
server_key_iv: "d4cf36ac2de815919ddf1eb5df3c63eb"
server_key_version: "encryptionkey@0"
I20260620 22:56:37.847616 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:37.861512 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:37.861846 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:37.861989 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:37.865917 7006 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:40255
I20260620 22:56:37.865942 7072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:40255 every 8 connection(s)
I20260620 22:56:37.866307 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:37.867376 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:37.871281 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3: Bootstrap starting.
I20260620 22:56:37.871966 7073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:37.872241 7073 log.cc:824] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:37.872831 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7006
I20260620 22:56:37.872949 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:37.873306 1688 external_mini_cluster.cc:1428] Setting key b50382be11739cb4cd9defc2d95a1488
I20260620 22:56:37.873349 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3: No bootstrap required, opened a new log
I20260620 22:56:37.876025 7073 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd512a64df894032b434c0fa2e9f5ac3" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40255 } }
I20260620 22:56:37.876183 7073 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:37.876216 7073 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd512a64df894032b434c0fa2e9f5ac3, State: Initialized, Role: FOLLOWER
I20260620 22:56:37.876350 7073 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd512a64df894032b434c0fa2e9f5ac3" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40255 } }
I20260620 22:56:37.876430 7073 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:37.876466 7073 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:37.876510 7073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:37.877290 7073 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd512a64df894032b434c0fa2e9f5ac3" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40255 } }
I20260620 22:56:37.877380 7073 leader_election.cc:302] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd512a64df894032b434c0fa2e9f5ac3; no voters:
I20260620 22:56:37.877528 7073 leader_election.cc:288] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:37.877568 7078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:37.877673 7078 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [term 1 LEADER]: Becoming Leader. State: Replica: cd512a64df894032b434c0fa2e9f5ac3, State: Running, Role: LEADER
I20260620 22:56:37.877753 7078 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd512a64df894032b434c0fa2e9f5ac3" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40255 } }
I20260620 22:56:37.877802 7073 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:37.878050 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd512a64df894032b434c0fa2e9f5ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd512a64df894032b434c0fa2e9f5ac3" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40255 } } }
I20260620 22:56:37.878118 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:37.878237 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd512a64df894032b434c0fa2e9f5ac3. Latest consensus state: current_term: 1 leader_uuid: "cd512a64df894032b434c0fa2e9f5ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd512a64df894032b434c0fa2e9f5ac3" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40255 } } }
I20260620 22:56:37.878288 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:37.878613 7082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:37.879158 7082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:37.879370 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.873868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.1.166.62:40255)
0620 22:56:37.874051 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.874058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.874083 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:56:37.874772 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.874779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.874846 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.874973 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:37.875232 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.876128 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.877135 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.877611 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.877736 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.878499 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:37.878503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:37.878504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:37.878714 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:37.878756 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:37.878772 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:37.878838 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:37.879052 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 22:56:37.879160 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260620 22:56:37.880610 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:37.881058 7082 catalog_manager.cc:1347] Generated new cluster ID: 29f1e949c88342a6afcf39dad0e3fa3e
I20260620 22:56:37.881119 7082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:37.896559 7082 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:37.897115 7082 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:37.906077 7082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd512a64df894032b434c0fa2e9f5ac3: Generated new TSK 0
I20260620 22:56:37.906409 7082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:37.915395 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:37.911795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.1.166.62:40255)
0620 22:56:37.911847 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:37.911850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:37.911960 (+ 110us) server_negotiation.cc:407] Connection header received
0620 22:56:37.912052 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:37.912053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:37.912067 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:37.912140 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:56:37.912271 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.912824 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.913635 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.913837 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.913897 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.914091 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:56:37.915143 (+ 1052us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 22:56:37.915294 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39478: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 22:56:37.915465 7077 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39478: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 22:56:37.915537 7102 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:37.911709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:40255 (local address 127.0.0.1:39478)
0620 22:56:37.911890 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:37.911902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:37.911973 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:37.912146 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:37.912147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:37.912148 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:56:37.912211 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:37.912214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.912922 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.912923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:37.913538 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:37.913540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:37.913917 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:37.913918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:37.913980 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:37.913988 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:56:37.915268 (+ 1280us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 22:56:37.915415 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.166.62:40255: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37487/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260620 22:56:37.916150 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7006
E20260620 22:56:37.926642 7103 webserver.cc:533] Webserver: sslize failed:
E20260620 22:56:37.926864 7103 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T22:56:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T22:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:37Z Disabled control of system clock
I20260620 22:56:37.962250 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:44217
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:44217
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/testchainca.cert with env {}
W20260620 22:56:38.053439 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:38.053789 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:38.053819 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:38.056034 7107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:38.056123 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:38.056134 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:38.056149 7107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:38.056161 7107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:56:38.056176 7107 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:56:38.056200 7107 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 22:56:38.058521 7107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:44217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:44217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:38.058797 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:38.059015 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:38.063822 7113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:38.063925 7107 server_base.cc:1034] running on GCE node
W20260620 22:56:38.063853 7115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:38.064080 7112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:38.064333 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:38.064659 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:38.065850 7107 hybrid_clock.cc:648] HybridClock initialized: now 1781996198065818 us; error 72 us; skew 500 ppm
I20260620 22:56:38.067348 7107 webserver.cc:460] Webserver started at http://127.1.166.62:35817/ using document root <none> and password file <none>
I20260620 22:56:38.067565 7107 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:38.067610 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:38.067701 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:38.068746 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "ec0390d7a14d421a8d61b50e48c2b193"
format_stamp: "Formatted at 2026-06-20 22:56:38 on dist-test-slave-rdl9"
server_key: "41041404b2db0d4d3d89105d809ad577"
server_key_iv: "f59b997a0f2dcd436642bf9d309a252d"
server_key_version: "encryptionkey@0"
I20260620 22:56:38.069059 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "ec0390d7a14d421a8d61b50e48c2b193"
format_stamp: "Formatted at 2026-06-20 22:56:38 on dist-test-slave-rdl9"
server_key: "41041404b2db0d4d3d89105d809ad577"
server_key_iv: "f59b997a0f2dcd436642bf9d309a252d"
server_key_version: "encryptionkey@0"
I20260620 22:56:38.070482 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:56:38.071653 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:38.071915 7107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:38.072044 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "ec0390d7a14d421a8d61b50e48c2b193"
format_stamp: "Formatted at 2026-06-20 22:56:38 on dist-test-slave-rdl9"
server_key: "41041404b2db0d4d3d89105d809ad577"
server_key_iv: "f59b997a0f2dcd436642bf9d309a252d"
server_key_version: "encryptionkey@0"
I20260620 22:56:38.072125 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:38.088577 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:38.088867 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:38.088970 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:38.093021 7107 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:44217
I20260620 22:56:38.093050 7173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:44217 every 8 connection(s)
I20260620 22:56:38.093369 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:38.094381 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:38.096797 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193: Bootstrap starting.
I20260620 22:56:38.097393 7174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:38.097402 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7107
I20260620 22:56:38.097491 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:38.097675 1688 external_mini_cluster.cc:1428] Setting key 6b2e3e2e98f1276717a33a77aab0ff5d
I20260620 22:56:38.097705 7174 log.cc:824] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:38.098644 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193: No bootstrap required, opened a new log
I20260620 22:56:38.101213 7174 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0390d7a14d421a8d61b50e48c2b193" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44217 } }
I20260620 22:56:38.101426 7174 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:38.101472 7174 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0390d7a14d421a8d61b50e48c2b193, State: Initialized, Role: FOLLOWER
I20260620 22:56:38.101606 7174 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0390d7a14d421a8d61b50e48c2b193" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44217 } }
I20260620 22:56:38.101689 7174 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:38.101713 7174 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:38.101761 7174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:38.102497 7174 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0390d7a14d421a8d61b50e48c2b193" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44217 } }
I20260620 22:56:38.102600 7174 leader_election.cc:302] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0390d7a14d421a8d61b50e48c2b193; no voters:
I20260620 22:56:38.102769 7174 leader_election.cc:288] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:38.102833 7179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:38.103004 7179 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [term 1 LEADER]: Becoming Leader. State: Replica: ec0390d7a14d421a8d61b50e48c2b193, State: Running, Role: LEADER
I20260620 22:56:38.103050 7174 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:38.103099 7179 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0390d7a14d421a8d61b50e48c2b193" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44217 } }
I20260620 22:56:38.103430 7179 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0390d7a14d421a8d61b50e48c2b193. Latest consensus state: current_term: 1 leader_uuid: "ec0390d7a14d421a8d61b50e48c2b193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0390d7a14d421a8d61b50e48c2b193" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44217 } } }
I20260620 22:56:38.103515 7179 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:38.103708 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0390d7a14d421a8d61b50e48c2b193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0390d7a14d421a8d61b50e48c2b193" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44217 } } }
I20260620 22:56:38.103808 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:38.103856 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:38.098200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.1.166.62:44217)
0620 22:56:38.098433 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:38.098441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:38.098469 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:56:38.099012 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:38.099019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:38.099082 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:38.099206 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:38.099486 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:38.100464 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:38.101574 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:38.102096 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:38.102208 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:38.102977 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:38.102981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:38.102982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:38.103278 (+ 296us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:38.103325 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:38.103343 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:38.103403 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:38.103507 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 22:56:38.103605 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260620 22:56:38.104271 7185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:38.104805 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:38.104856 7185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:38.106503 7185 catalog_manager.cc:1347] Generated new cluster ID: 4caba51a288f4895bd3287938970ae1c
I20260620 22:56:38.106559 7185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:38.129202 7185 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:38.129783 7185 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:38.159698 7185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0390d7a14d421a8d61b50e48c2b193: Generated new TSK 0
I20260620 22:56:38.159946 7185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:38.279363 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:38.170570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.1.166.62:44217)
0620 22:56:38.170626 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:38.170628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:38.170724 (+ 96us) server_negotiation.cc:407] Connection header received
0620 22:56:38.170803 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:38.170804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:38.170814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:38.170876 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:56:38.171025 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:38.171521 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:38.172746 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:38.172991 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:38.173095 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:38.173449 (+ 354us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:56:38.278896 (+105447us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 22:56:38.279291 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41390: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 22:56:38.279438 7178 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41390: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 22:56:38.279434 7203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:38.170480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:44217 (local address 127.0.0.1:41390)
0620 22:56:38.170631 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:38.170644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:38.170727 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:38.170894 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:38.170895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:38.170897 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:56:38.170964 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:38.170967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:38.171640 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:38.171642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:38.172542 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:38.172548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:38.173137 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:38.173139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:38.173216 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:38.173225 (+ 9us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:56:38.279206 (+105981us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 22:56:38.279350 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.166.62:44217: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260620 22:56:38.279731 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7107
2026-06-20T22:56:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (342 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (94 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (104 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:38 dist-test-slave-rdl9 krb5kdc[7215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:38 dist-test-slave-rdl9 krb5kdc[7215](info): set up 2 sockets
Jun 20 22:56:38 dist-test-slave-rdl9 krb5kdc[7215](info): commencing operation
krb5kdc: starting...
W20260620 22:56:39.450942 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:39 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:56:39.578459 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:40373
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:40373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:39.673101 7228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:39.673627 7228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:39.673780 7228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:39.675251 7228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:39.675323 7228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:39.675346 7228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:39.675367 7228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:39.675388 7228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:39.677083 7228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:40373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:40373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:39.677387 7228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:39.677624 7228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:39.682627 7234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:39.682611 7236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:39.682614 7233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:39.682950 7228 server_base.cc:1034] running on GCE node
I20260620 22:56:39.683293 7228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:39.683632 7228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:39.684819 7228 hybrid_clock.cc:648] HybridClock initialized: now 1781996199684798 us; error 34 us; skew 500 ppm
Jun 20 22:56:39 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:39.687741 7228 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:56:39.688401 7228 webserver.cc:460] Webserver started at http://127.1.166.62:34407/ using document root <none> and password file <none>
I20260620 22:56:39.688647 7228 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:39.688704 7228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:39.688844 7228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:39.690146 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "b29f0a922e5d44bcb1016f251716e066"
format_stamp: "Formatted at 2026-06-20 22:56:39 on dist-test-slave-rdl9"
server_key: "91bab98a8cccd371101f9ae3420ba27d"
server_key_iv: "854b2b9a9384ab9f025c3253648d2f6b"
server_key_version: "encryptionkey@0"
I20260620 22:56:39.690582 7228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "b29f0a922e5d44bcb1016f251716e066"
format_stamp: "Formatted at 2026-06-20 22:56:39 on dist-test-slave-rdl9"
server_key: "91bab98a8cccd371101f9ae3420ba27d"
server_key_iv: "854b2b9a9384ab9f025c3253648d2f6b"
server_key_version: "encryptionkey@0"
I20260620 22:56:39.692060 7228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:39.693097 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:39.693324 7228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:39.693424 7228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "b29f0a922e5d44bcb1016f251716e066"
format_stamp: "Formatted at 2026-06-20 22:56:39 on dist-test-slave-rdl9"
server_key: "91bab98a8cccd371101f9ae3420ba27d"
server_key_iv: "854b2b9a9384ab9f025c3253648d2f6b"
server_key_version: "encryptionkey@0"
I20260620 22:56:39.693497 7228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:39.706073 7228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:39.708586 7228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:39.708745 7228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:39.713874 7228 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:40373
I20260620 22:56:39.713968 7295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:40373 every 8 connection(s)
I20260620 22:56:39.714323 7228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:39.715240 7296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:39.717556 7296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066: Bootstrap starting.
I20260620 22:56:39.718130 7296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:39.718444 7296 log.cc:824] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:39.719177 7296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066: No bootstrap required, opened a new log
I20260620 22:56:39.720757 7296 raft_consensus.cc:348] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29f0a922e5d44bcb1016f251716e066" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40373 } }
I20260620 22:56:39.720911 7296 raft_consensus.cc:374] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:39.720934 7296 raft_consensus.cc:729] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29f0a922e5d44bcb1016f251716e066, State: Initialized, Role: FOLLOWER
I20260620 22:56:39.721062 7296 consensus_queue.cc:260] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29f0a922e5d44bcb1016f251716e066" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40373 } }
I20260620 22:56:39.721131 7296 raft_consensus.cc:388] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:39.721164 7296 raft_consensus.cc:482] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:39.721211 7296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:39.721752 7296 raft_consensus.cc:504] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29f0a922e5d44bcb1016f251716e066" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40373 } }
I20260620 22:56:39.721848 7296 leader_election.cc:302] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b29f0a922e5d44bcb1016f251716e066; no voters:
I20260620 22:56:39.722003 7296 leader_election.cc:288] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:39.722091 7299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:39.722261 7296 sys_catalog.cc:564] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:39.722486 7299 raft_consensus.cc:686] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [term 1 LEADER]: Becoming Leader. State: Replica: b29f0a922e5d44bcb1016f251716e066, State: Running, Role: LEADER
I20260620 22:56:39.722589 7299 consensus_queue.cc:237] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29f0a922e5d44bcb1016f251716e066" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40373 } }
I20260620 22:56:39.723094 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b29f0a922e5d44bcb1016f251716e066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29f0a922e5d44bcb1016f251716e066" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40373 } } }
I20260620 22:56:39.723201 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:39.723098 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b29f0a922e5d44bcb1016f251716e066. Latest consensus state: current_term: 1 leader_uuid: "b29f0a922e5d44bcb1016f251716e066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29f0a922e5d44bcb1016f251716e066" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 40373 } } }
I20260620 22:56:39.723390 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:39.723405 7308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:39.723920 7308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:39.724201 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7228
I20260620 22:56:39.724285 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:39.724457 1688 external_mini_cluster.cc:1428] Setting key bb9093a0a6e6f95b3a35b0c968218857
I20260620 22:56:39.725647 7308 catalog_manager.cc:1347] Generated new cluster ID: 21a370c6aff048f0bdb1cc2d8a888273
I20260620 22:56:39.725704 7308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:56:39 dist-test-slave-rdl9 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:39.734596 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:39.725062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.1.166.62:40373)
0620 22:56:39.725227 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:39.725237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:39.725268 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:56:39.725389 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:39.725397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:39.725455 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:39.727172 (+ 1717us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:39.727257 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:39.728197 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:39.728939 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:39.729379 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:39.729484 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:39.731680 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:39.731684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:39.731685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:39.731697 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:39.733157 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:39.733466 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:39.733469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:39.733470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:39.733562 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:39.733729 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:39.733730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:39.733730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:39.733981 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:39.734130 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:39.734226 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 22:56:39.734397 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260620 22:56:39.745198 7308 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:39.745775 7308 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:39.748262 7308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b29f0a922e5d44bcb1016f251716e066: Generated new TSK 0
I20260620 22:56:39.748454 7308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:56:39.781133 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:40373
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:39.898895 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:39.899158 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:39.899184 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:39.900507 7322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:39.900574 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:39.900606 7322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:39.902310 7322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:40373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:39.902643 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:39.902846 7322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:39.908304 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:39.908306 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:39.908375 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:39.908667 7322 server_base.cc:1034] running on GCE node
I20260620 22:56:39.908905 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:39.909178 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:39.910390 7322 hybrid_clock.cc:648] HybridClock initialized: now 1781996199910362 us; error 43 us; skew 500 ppm
Jun 20 22:56:39 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:39.913525 7322 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:56:39.914177 7322 webserver.cc:460] Webserver started at http://127.1.166.1:41315/ using document root <none> and password file <none>
I20260620 22:56:39.914500 7322 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:39.914561 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:39.914710 7322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:39.916139 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "406960be285b47219be530d313a3b1a6"
format_stamp: "Formatted at 2026-06-20 22:56:39 on dist-test-slave-rdl9"
server_key: "8e24d1ab0c89f8646b4ff8ef39cc0d7b"
server_key_iv: "7b695e9293f3d0c718ecea55d6a93578"
server_key_version: "encryptionkey@0"
I20260620 22:56:39.916580 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "406960be285b47219be530d313a3b1a6"
format_stamp: "Formatted at 2026-06-20 22:56:39 on dist-test-slave-rdl9"
server_key: "8e24d1ab0c89f8646b4ff8ef39cc0d7b"
server_key_iv: "7b695e9293f3d0c718ecea55d6a93578"
server_key_version: "encryptionkey@0"
I20260620 22:56:39.918090 7322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:39.919016 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:39.919203 7322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:56:39.919296 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "406960be285b47219be530d313a3b1a6"
format_stamp: "Formatted at 2026-06-20 22:56:39 on dist-test-slave-rdl9"
server_key: "8e24d1ab0c89f8646b4ff8ef39cc0d7b"
server_key_iv: "7b695e9293f3d0c718ecea55d6a93578"
server_key_version: "encryptionkey@0"
I20260620 22:56:39.919374 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:39.946422 7322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:39.948637 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:39.948782 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:39.949023 7322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:39.949334 7322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:39.949368 7322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:39.949397 7322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:39.949420 7322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:39.955123 7322 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:39483
I20260620 22:56:39.955155 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:39483 every 8 connection(s)
I20260620 22:56:39.955531 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:56:39.957705 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7322
I20260620 22:56:39.957808 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:39.957966 1688 external_mini_cluster.cc:1428] Setting key a40efb8126a3d24e4165d2c513e62751
Jun 20 22:56:39 dist-test-slave-rdl9 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
I20260620 22:56:39.965553 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:39.956340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:38977 (local address 127.1.166.62:40373)
0620 22:56:39.956483 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:39.956486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:39.956645 (+ 159us) server_negotiation.cc:407] Connection header received
0620 22:56:39.957598 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:39.957600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:39.957614 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:39.957667 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:39.958453 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:39.958995 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:39.960153 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:39.960358 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:39.960416 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:39.962677 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:39.962680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:39.962680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:39.962695 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:39.964069 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:39.964533 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:39.964535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:39.964535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:39.964590 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:39.964835 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:39.964836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:39.964836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:39.964961 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:39.965022 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:39.965342 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 22:56:39.965455 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 22:56:39.965725 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:39.956258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:40373 (local address 127.1.166.1:38977)
0620 22:56:39.956512 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:39.956558 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:39.957490 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:39.957738 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:39.957742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:39.958085 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:39.958370 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:39.958377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:39.959085 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:39.959085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:39.960066 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:39.960070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:39.960442 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:39.960442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:39.960573 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:39.961076 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:39.961085 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:39.962561 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:39.964172 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:39.964186 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:39.964189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:39.964456 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:39.964650 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:39.964651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:39.964652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:39.964776 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:39.965020 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:39.965027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:39.965209 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 22:56:39.965310 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 22:56:39.965490 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260620 22:56:39.966212 7451 heartbeater.cc:347] Connected to a master server at 127.1.166.62:40373
I20260620 22:56:39.966323 7451 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:39.966599 7451 heartbeater.cc:511] Master 127.1.166.62:40373 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
I20260620 22:56:39.967258 7260 ts_manager.cc:194] Registered new tserver with Master: 406960be285b47219be530d313a3b1a6 (127.1.166.1:39483)
I20260620 22:56:39.968135 7260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:38977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:56:39.984611 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:40373
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:40.077064 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:40.077330 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:40.077359 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:40.078670 7456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:40.078716 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:40.078745 7456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:40.080389 7456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:40373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:40.080626 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:40.080807 7456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:40.086265 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:40.086266 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:40.086282 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:40.086592 7456 server_base.cc:1034] running on GCE node
I20260620 22:56:40.086823 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:40.087070 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:40.088238 7456 hybrid_clock.cc:648] HybridClock initialized: now 1781996200088209 us; error 46 us; skew 500 ppm
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.091001 7456 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:56:40.091454 7456 webserver.cc:460] Webserver started at http://127.1.166.2:46683/ using document root <none> and password file <none>
I20260620 22:56:40.091637 7456 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:40.091678 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:40.091769 7456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:40.092842 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "68ea96fbab414841acd16bf66c23d182"
format_stamp: "Formatted at 2026-06-20 22:56:40 on dist-test-slave-rdl9"
server_key: "8588eeb2f83dad6316fff1dbe954f07f"
server_key_iv: "3ade76adb12219f014fffb116706e096"
server_key_version: "encryptionkey@0"
I20260620 22:56:40.093171 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "68ea96fbab414841acd16bf66c23d182"
format_stamp: "Formatted at 2026-06-20 22:56:40 on dist-test-slave-rdl9"
server_key: "8588eeb2f83dad6316fff1dbe954f07f"
server_key_iv: "3ade76adb12219f014fffb116706e096"
server_key_version: "encryptionkey@0"
I20260620 22:56:40.094594 7456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:40.095322 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:40.095474 7456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:40.095558 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "68ea96fbab414841acd16bf66c23d182"
format_stamp: "Formatted at 2026-06-20 22:56:40 on dist-test-slave-rdl9"
server_key: "8588eeb2f83dad6316fff1dbe954f07f"
server_key_iv: "3ade76adb12219f014fffb116706e096"
server_key_version: "encryptionkey@0"
I20260620 22:56:40.095629 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:40.106671 7456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:40.108940 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:40.109086 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:40.109313 7456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:40.109611 7456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:40.109643 7456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:40.109673 7456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:40.109696 7456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:40.116012 7456 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:34723
I20260620 22:56:40.116061 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:34723 every 8 connection(s)
I20260620 22:56:40.116381 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:40.120250 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7456
I20260620 22:56:40.120354 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:40.120532 1688 external_mini_cluster.cc:1428] Setting key afa2c498d21787493cd5dbf1c37eda55
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:40.126184 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:40.117077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:56031 (local address 127.1.166.62:40373)
0620 22:56:40.117161 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:40.117164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:40.117327 (+ 163us) server_negotiation.cc:407] Connection header received
0620 22:56:40.118273 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:40.118274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:40.118288 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:40.118366 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:40.119029 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.119527 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.120821 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.121031 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.121098 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:40.123529 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:40.123531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:40.123531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:40.123538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:40.124858 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.125307 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.125308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.125309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.125361 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.125578 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.125578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.125579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.125687 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:40.125743 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:40.126002 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 22:56:40.126092 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:56:40.126392 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:40.117016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:40373 (local address 127.1.166.2:56031)
0620 22:56:40.117199 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:40.117243 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:40.118160 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:40.118482 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:40.118485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:40.118745 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:40.118940 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:40.118944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.119605 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.119606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:40.120618 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:40.120621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.121142 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.121143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:40.121286 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:40.121820 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:40.121830 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:40.123422 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:40.124999 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:40.125007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:40.125009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:40.125237 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:40.125414 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:40.125415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:40.125415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:40.125523 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:40.125740 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:40.125745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:40.125887 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 22:56:40.125982 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 22:56:40.126143 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260620 22:56:40.126827 7585 heartbeater.cc:347] Connected to a master server at 127.1.166.62:40373
I20260620 22:56:40.126952 7585 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:40.127154 7585 heartbeater.cc:511] Master 127.1.166.62:40373 requested a full tablet report, sending...
I20260620 22:56:40.127599 7260 ts_manager.cc:194] Registered new tserver with Master: 68ea96fbab414841acd16bf66c23d182 (127.1.166.2:34723)
I20260620 22:56:40.127982 7260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:56031
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:56:40.145079 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:40373
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:40.233978 7590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:40.234233 7590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:40.234261 7590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:40.235831 7590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:40.235914 7590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:40.235948 7590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:40.237658 7590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:40373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:40.237901 7590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:40.238097 7590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:40.243153 7598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:40.243209 7596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:40.243155 7595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:40.243649 7590 server_base.cc:1034] running on GCE node
I20260620 22:56:40.243887 7590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:40.244144 7590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:40.245311 7590 hybrid_clock.cc:648] HybridClock initialized: now 1781996200245298 us; error 35 us; skew 500 ppm
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.248255 7590 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:56:40.248864 7590 webserver.cc:460] Webserver started at http://127.1.166.3:43791/ using document root <none> and password file <none>
I20260620 22:56:40.249104 7590 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:40.249156 7590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:40.249286 7590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:40.250489 7590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "df9ba425c4c040abbcc281595cf59ec7"
format_stamp: "Formatted at 2026-06-20 22:56:40 on dist-test-slave-rdl9"
server_key: "b87a42b90357bdfc211edd4249e22aa7"
server_key_iv: "070df7164f68da87d274e8ca2728534c"
server_key_version: "encryptionkey@0"
I20260620 22:56:40.250875 7590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "df9ba425c4c040abbcc281595cf59ec7"
format_stamp: "Formatted at 2026-06-20 22:56:40 on dist-test-slave-rdl9"
server_key: "b87a42b90357bdfc211edd4249e22aa7"
server_key_iv: "070df7164f68da87d274e8ca2728534c"
server_key_version: "encryptionkey@0"
I20260620 22:56:40.252404 7590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:40.253566 7605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:40.253818 7590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:40.253944 7590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "df9ba425c4c040abbcc281595cf59ec7"
format_stamp: "Formatted at 2026-06-20 22:56:40 on dist-test-slave-rdl9"
server_key: "b87a42b90357bdfc211edd4249e22aa7"
server_key_iv: "070df7164f68da87d274e8ca2728534c"
server_key_version: "encryptionkey@0"
I20260620 22:56:40.254036 7590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:40.263520 7590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:40.265952 7590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:40.266109 7590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:40.266450 7590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:40.266809 7590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:40.266846 7590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:40.266878 7590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:40.266896 7590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:40.272964 7590 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:39843
I20260620 22:56:40.273078 7718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:39843 every 8 connection(s)
I20260620 22:56:40.273438 7590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:40.280992 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7590
I20260620 22:56:40.281073 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:56:40.281224 1688 external_mini_cluster.cc:1428] Setting key 92506893297d97d60b34f76863c8008d
I20260620 22:56:40.284989 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:40.274521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:48631 (local address 127.1.166.62:40373)
0620 22:56:40.274582 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:40.274584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:40.274760 (+ 176us) server_negotiation.cc:407] Connection header received
0620 22:56:40.275788 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:40.275790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:40.275804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:40.275886 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:40.276702 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.277419 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.278678 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.278902 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.278972 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:40.281887 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:40.281889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:40.281889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:40.281901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:40.283542 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.284035 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.284036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.284036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.284099 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.284335 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.284336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.284336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.284462 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:40.284518 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:40.284809 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 22:56:40.284894 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:56:40.285199 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:40.274423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:40373 (local address 127.1.166.3:48631)
0620 22:56:40.274599 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:40.274647 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:40.275644 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:40.276002 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:40.276007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:40.276306 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:40.276547 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:40.276552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.277564 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.277565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:40.278562 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:40.278565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.279038 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.279040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:40.279208 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:40.279913 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:40.279934 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:40.281763 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:40.283647 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:40.283662 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:40.283665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:40.283960 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:40.284154 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:40.284155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:40.284155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:40.284273 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:40.284518 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:40.284525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:40.284679 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 22:56:40.284773 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:56:40.284968 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260620 22:56:40.285686 7719 heartbeater.cc:347] Connected to a master server at 127.1.166.62:40373
I20260620 22:56:40.285768 7719 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:40.285952 7719 heartbeater.cc:511] Master 127.1.166.62:40373 requested a full tablet report, sending...
I20260620 22:56:40.286369 7260 ts_manager.cc:194] Registered new tserver with Master: df9ba425c4c040abbcc281595cf59ec7 (127.1.166.3:39843)
I20260620 22:56:40.286763 7260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:48631
I20260620 22:56:40.294242 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/test-admin.keytab.
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.308696 1688 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 22:56:40.310257 7726 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:40.309572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.1.166.1:39483)
0620 22:56:40.309724 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:40.309728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:40.309739 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:40.309766 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:40.309767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:40.309780 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:40.309849 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:40.310179 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45086: BlockingRecv error: recv got EOF from 127.0.0.1:45086 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
W20260620 22:56:40.310495 1688 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc)
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.311417 1688 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.312213 1688 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:40.321797 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:40.312837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.1.166.1:39483)
0620 22:56:40.312912 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:40.312914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:40.312932 (+ 18us) server_negotiation.cc:407] Connection header received
0620 22:56:40.313031 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:40.313032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:40.313050 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:40.313103 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:40.313777 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.314883 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.315730 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.316090 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.316154 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:40.318951 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:40.318963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:40.318963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:40.318972 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:40.320602 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.320997 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.321000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.321001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.321055 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.321219 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.321220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.321220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.321367 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:40.321514 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:40.321596 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 22:56:40.321686 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 22:56:40.323985 7726 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:40.323290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.1.166.1:39483)
0620 22:56:40.323336 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:40.323337 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:40.323362 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:56:40.323455 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:40.323456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:40.323468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:40.323524 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:40.323929 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45094: BlockingRecv error: recv got EOF from 127.0.0.1:45094 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:56:40.324085 1688 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc' of type FILE
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.325457 1688 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.326428 1688 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:40.333433 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:40.326953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.1.166.1:39483)
0620 22:56:40.327019 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:40.327021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:40.327043 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:56:40.327094 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:40.327095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:40.327104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:40.327140 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:40.327500 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.328005 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.328688 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.328904 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.328964 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:40.331102 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:40.331103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:40.331103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:40.331111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:40.332275 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.332633 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.332634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.332634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.332693 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.332897 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.332898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.332898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.333000 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:40.333073 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:40.333208 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:56:40.333336 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 22:56:40.335044 7726 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:40.334528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.1.166.1:39483)
0620 22:56:40.334582 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:40.334583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:40.334611 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:56:40.334677 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:40.334678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:40.334686 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:40.334728 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:40.334993 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45102: BlockingRecv error: recv got EOF from 127.0.0.1:45102 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 22:56:40.335129 1688 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996175763767-1688-0/krb5kdc/krb5cc)
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.336109 1688 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:40.336941 1688 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.1@KRBTEST.COM
I20260620 22:56:40.344055 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:40.337419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.1.166.1:39483)
0620 22:56:40.337459 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:40.337461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:40.337510 (+ 49us) server_negotiation.cc:407] Connection header received
0620 22:56:40.337549 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:40.337550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:40.337557 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:40.337599 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:40.337893 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.338418 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.339081 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:40.339274 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:40.339328 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:40.341365 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:40.341367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:40.341367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:40.341375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:40.342998 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.343349 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.343350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.343350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.343404 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:40.343560 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:40.343561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:40.343561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:40.343676 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:40.343762 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:40.343879 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:56:40.343958 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:56:40.344995 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7322
I20260620 22:56:40.349324 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7456
I20260620 22:56:40.353435 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7590
I20260620 22:56:40.357832 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7228
2026-06-20T22:56:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1880 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7739](info): set up 2 sockets
Jun 20 22:56:40 dist-test-slave-rdl9 krb5kdc[7739](info): commencing operation
krb5kdc: starting...
W20260620 22:56:41.331058 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:56:41 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.62.keytab.
Entry for principal oryx/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.62.keytab.
I20260620 22:56:41.453984 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:33827
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.62.keytab
--principal=oryx/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:33827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:41.536478 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:41.536725 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:41.536760 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:41.538029 7752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:41.538089 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:41.538107 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:41.538123 7752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:41.538139 7752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:41.539872 7752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:33827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.62.keytab
--principal=oryx/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:41.540166 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:41.540393 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:41.545464 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:41.545466 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:41.545676 7752 server_base.cc:1034] running on GCE node
W20260620 22:56:41.545464 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:41.546108 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:41.546482 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:41.547684 7752 hybrid_clock.cc:648] HybridClock initialized: now 1781996201547668 us; error 34 us; skew 500 ppm
Jun 20 22:56:41 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:41.550784 7752 init.cc:377] Logged in from keytab as oryx/127.1.166.62@KRBTEST.COM (short username oryx)
I20260620 22:56:41.551352 7752 webserver.cc:460] Webserver started at http://127.1.166.62:38129/ using document root <none> and password file <none>
I20260620 22:56:41.551584 7752 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:41.551638 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:41.551774 7752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:41.553006 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "90020d0371ed4eaf9ef69a85170700a2"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "8f31b98435f49aa9ef738e67c286c1f3"
server_key_iv: "face639a84c7d7dbb033826e9b616e7c"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.553392 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "90020d0371ed4eaf9ef69a85170700a2"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "8f31b98435f49aa9ef738e67c286c1f3"
server_key_iv: "face639a84c7d7dbb033826e9b616e7c"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.555158 7752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 22:56:41.555994 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:41.556161 7752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:41.556253 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "90020d0371ed4eaf9ef69a85170700a2"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "8f31b98435f49aa9ef738e67c286c1f3"
server_key_iv: "face639a84c7d7dbb033826e9b616e7c"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.556327 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:41.573745 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:41.575994 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:41.576138 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:41.580193 7752 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:33827
I20260620 22:56:41.580308 7819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:33827 every 8 connection(s)
I20260620 22:56:41.580612 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:41.581601 7820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:41.584048 7820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: Bootstrap starting.
I20260620 22:56:41.584632 7820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:41.584901 7820 log.cc:824] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:41.585703 7820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: No bootstrap required, opened a new log
I20260620 22:56:41.588132 7820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } }
I20260620 22:56:41.588337 7820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:41.588374 7820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90020d0371ed4eaf9ef69a85170700a2, State: Initialized, Role: FOLLOWER
I20260620 22:56:41.588518 7820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } }
I20260620 22:56:41.588595 7820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:41.588635 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:41.588681 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:41.589758 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } }
I20260620 22:56:41.589881 7820 leader_election.cc:302] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90020d0371ed4eaf9ef69a85170700a2; no voters:
I20260620 22:56:41.590051 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:41.590095 7823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:41.590317 7820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:41.590358 7823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 1 LEADER]: Becoming Leader. State: Replica: 90020d0371ed4eaf9ef69a85170700a2, State: Running, Role: LEADER
I20260620 22:56:41.590445 7823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } }
I20260620 22:56:41.590771 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90020d0371ed4eaf9ef69a85170700a2. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } }
I20260620 22:56:41.590845 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:41.591046 7830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:41.591565 7830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:41.591758 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7752
I20260620 22:56:41.591876 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:56:41.592149 1688 external_mini_cluster.cc:1428] Setting key a51b93ae1fdeb083c559a44de8acebd9
I20260620 22:56:41.590804 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } }
I20260620 22:56:41.592828 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:41.593302 7840 mvcc.cc:205] Tried to move back new op lower bound from 7299056441723035648 to 7299056441714872320. Current Snapshot: MvccSnapshot[applied={T|T < 7299056441723035648}]
I20260620 22:56:41.593780 7830 catalog_manager.cc:1347] Generated new cluster ID: e6d30f27358c40418885647fcbc0c04b
I20260620 22:56:41.593834 7830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:56:41 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:56:41.600419 7830 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:41.601107 7830 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:41.602700 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:41.593166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.1.166.62:33827)
0620 22:56:41.593441 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:41.593449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:41.593642 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:56:41.593765 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:41.593772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:41.593831 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:41.593975 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:41.594578 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.595494 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.596225 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.596651 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.596748 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:41.599058 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:41.599063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:41.599063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:41.599075 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:41.601275 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:41.601598 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:41.601602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:41.601602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:41.601699 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:41.601867 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:41.601868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:41.601868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:41.602085 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:41.602241 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:41.602313 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 22:56:41.602508 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":78,"threads_started":1}
I20260620 22:56:41.609822 7830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: Generated new TSK 0
I20260620 22:56:41.610051 7830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab.
Entry for principal oryx/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab.
I20260620 22:56:41.641376 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:33827
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab
--principal=oryx/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:41.730818 7846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:41.731070 7846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:41.731109 7846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:41.732416 7846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:41.732476 7846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:41.732517 7846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:41.734135 7846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab
--principal=oryx/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:41.734429 7846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:41.734661 7846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:41.739547 7854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:41.739602 7852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:41.739817 7846 server_base.cc:1034] running on GCE node
W20260620 22:56:41.739838 7851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:41.740150 7846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:41.740439 7846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:41.741606 7846 hybrid_clock.cc:648] HybridClock initialized: now 1781996201741593 us; error 37 us; skew 500 ppm
Jun 20 22:56:41 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:41.744915 7846 init.cc:377] Logged in from keytab as oryx/127.1.166.1@KRBTEST.COM (short username oryx)
I20260620 22:56:41.745771 7846 webserver.cc:460] Webserver started at http://127.1.166.1:44315/ using document root <none> and password file <none>
I20260620 22:56:41.746042 7846 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:41.746096 7846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:41.746232 7846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:41.747370 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "550e61ba7c3542bb9b06f7c1baaf87a2"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "dd10680d420e797360e0927fe87aaa5d"
server_key_iv: "9acb0fd2cbbee8589ca538e8fb79a077"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.747725 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "550e61ba7c3542bb9b06f7c1baaf87a2"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "dd10680d420e797360e0927fe87aaa5d"
server_key_iv: "9acb0fd2cbbee8589ca538e8fb79a077"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.749337 7846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:41.750406 7861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:41.750591 7846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:41.750675 7846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "550e61ba7c3542bb9b06f7c1baaf87a2"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "dd10680d420e797360e0927fe87aaa5d"
server_key_iv: "9acb0fd2cbbee8589ca538e8fb79a077"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.750735 7846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:41.765170 7846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:41.767356 7846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:41.767477 7846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:41.767679 7846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:41.768046 7846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:41.768095 7846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:41.768119 7846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:41.768132 7846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:41.773964 7846 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:38073
I20260620 22:56:41.774035 7974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:38073 every 8 connection(s)
I20260620 22:56:41.774399 7846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:56:41.778236 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7846
I20260620 22:56:41.778398 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:56:41.778601 1688 external_mini_cluster.cc:1428] Setting key f73a4227682453594acab855c2508077
Jun 20 22:56:41 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:56:41.786129 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:41.775481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:44907 (local address 127.1.166.62:33827)
0620 22:56:41.775592 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:41.775595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:41.775837 (+ 242us) server_negotiation.cc:407] Connection header received
0620 22:56:41.776344 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:41.776346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:41.776364 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:41.776436 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:41.777570 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.778106 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.779540 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.779763 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.779828 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:41.782271 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:41.782273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:41.782273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:41.782283 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:41.783851 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:41.784633 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:41.784635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:41.784636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:41.784717 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:41.785132 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:41.785133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:41.785134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:41.785323 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:41.785403 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:41.785809 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 22:56:41.785982 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 22:56:41.786218 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:41.775404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:44907)
0620 22:56:41.775673 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:41.775718 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:41.776211 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:41.776515 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:41.776520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:41.777160 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:41.777377 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:41.777381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.778235 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.778237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:41.779453 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:41.779457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.779856 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.779858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:41.780014 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:41.780578 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:41.780589 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:41.782147 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:41.784017 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:41.784037 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:41.784041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:41.784428 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:41.784824 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:41.784825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:41.784826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:41.785029 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:41.785419 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:41.785427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:41.785608 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 22:56:41.785737 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 22:56:41.785962 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":109,"threads_started":1}
I20260620 22:56:41.786948 7975 heartbeater.cc:347] Connected to a master server at 127.1.166.62:33827
I20260620 22:56:41.787055 7975 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:41.787250 7975 heartbeater.cc:511] Master 127.1.166.62:33827 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.1.166.2@KRBTEST.COM; defaulting to no policy
I20260620 22:56:41.787781 7784 ts_manager.cc:194] Registered new tserver with Master: 550e61ba7c3542bb9b06f7c1baaf87a2 (127.1.166.1:38073)
I20260620 22:56:41.788434 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:44907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab.
Entry for principal oryx/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab.
I20260620 22:56:41.803527 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:33827
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab
--principal=oryx/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:41.893476 7980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:41.893855 7980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:41.893895 7980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:41.895381 7980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:41.895447 7980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:41.895483 7980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:41.897262 7980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab
--principal=oryx/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:41.897549 7980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:41.897823 7980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:41.903352 7988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:41.903358 7986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:41.903357 7985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:41.903723 7980 server_base.cc:1034] running on GCE node
I20260620 22:56:41.904197 7980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:41.904553 7980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:41.905750 7980 hybrid_clock.cc:648] HybridClock initialized: now 1781996201905729 us; error 42 us; skew 500 ppm
Jun 20 22:56:41 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:41.908983 7980 init.cc:377] Logged in from keytab as oryx/127.1.166.2@KRBTEST.COM (short username oryx)
I20260620 22:56:41.909839 7980 webserver.cc:460] Webserver started at http://127.1.166.2:39809/ using document root <none> and password file <none>
I20260620 22:56:41.910120 7980 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:41.910174 7980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:41.910311 7980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:41.911538 7980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "42fc7093901745a19824662cacddb9ff"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "0a50a71585533a24036d537ebcbf415e"
server_key_iv: "3f6d3f1d8a439527f1abe2c477d67c6b"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.911898 7980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "42fc7093901745a19824662cacddb9ff"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "0a50a71585533a24036d537ebcbf415e"
server_key_iv: "3f6d3f1d8a439527f1abe2c477d67c6b"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.913393 7980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:41.914490 7995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:41.914685 7980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:41.914777 7980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "42fc7093901745a19824662cacddb9ff"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "0a50a71585533a24036d537ebcbf415e"
server_key_iv: "3f6d3f1d8a439527f1abe2c477d67c6b"
server_key_version: "encryptionkey@0"
I20260620 22:56:41.914848 7980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:41.923837 7980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:41.926476 7980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:41.926628 7980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:41.926847 7980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:41.927191 7980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:41.927229 7980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:41.927256 7980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:41.927295 7980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:41.934181 7980 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:41901
I20260620 22:56:41.934271 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:41901 every 8 connection(s)
I20260620 22:56:41.934617 7980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:41.938915 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 7980
I20260620 22:56:41.939034 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:56:41.939199 1688 external_mini_cluster.cc:1428] Setting key 207a8d3faf79100e2947795496956b74
Jun 20 22:56:41 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:56:41.945588 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:41.935402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:43815 (local address 127.1.166.62:33827)
0620 22:56:41.935486 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:41.935488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:41.935756 (+ 268us) server_negotiation.cc:407] Connection header received
0620 22:56:41.936293 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:41.936294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:41.936307 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:41.936360 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:41.937379 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.937884 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.939000 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.939228 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.939288 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:41.941953 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:41.941955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:41.941956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:41.941965 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:41.943814 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:41.944409 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:41.944412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:41.944413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:41.944486 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:41.944832 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:41.944833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:41.944833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:41.944994 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:41.945069 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:41.945389 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 22:56:41.945484 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 22:56:41.945926 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:41.935344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:43815)
0620 22:56:41.935627 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:41.935668 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:41.936189 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:41.936436 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:41.936440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:41.937043 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:41.937271 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:41.937276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.937985 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.937986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:41.938903 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:41.938907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:41.939330 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:41.939331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:41.939455 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:41.940063 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:41.940074 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:41.941819 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:41.943910 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:41.943923 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:41.943926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:41.944197 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:41.944626 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:41.944627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:41.944628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:41.944759 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:41.945084 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:41.945091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:41.945246 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 22:56:41.945347 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 22:56:41.945502 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":192,"threads_started":1}
I20260620 22:56:41.946507 8109 heartbeater.cc:347] Connected to a master server at 127.1.166.62:33827
I20260620 22:56:41.946693 8109 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:41.946933 8109 heartbeater.cc:511] Master 127.1.166.62:33827 requested a full tablet report, sending...
I20260620 22:56:41.947397 7784 ts_manager.cc:194] Registered new tserver with Master: 42fc7093901745a19824662cacddb9ff (127.1.166.2:41901)
I20260620 22:56:41.947800 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:43815
WARNING: no policy specified for oryx/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab.
Entry for principal oryx/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab.
I20260620 22:56:41.966104 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:33827
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab
--principal=oryx/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:42.050815 8114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:42.051039 8114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:42.051064 8114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:42.052333 8114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:42.052390 8114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:42.052421 8114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:42.054103 8114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab
--principal=oryx/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:42.054369 8114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:42.054600 8114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:42.059985 8119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.060019 8120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.060089 8114 server_base.cc:1034] running on GCE node
W20260620 22:56:42.059989 8122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.060459 8114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:42.060757 8114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:42.061908 8114 hybrid_clock.cc:648] HybridClock initialized: now 1781996202061869 us; error 57 us; skew 500 ppm
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:42.064751 8114 init.cc:377] Logged in from keytab as oryx/127.1.166.3@KRBTEST.COM (short username oryx)
I20260620 22:56:42.065270 8114 webserver.cc:460] Webserver started at http://127.1.166.3:35543/ using document root <none> and password file <none>
I20260620 22:56:42.065480 8114 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.065526 8114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.065634 8114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:42.066795 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "2e1a552656224c9facb4281c4d0de28d"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "c67669ba4ef7e92c6b2c10cb1df0bfaf"
server_key_iv: "2e7a9b693c497391393fbb1613ca902d"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.067126 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "2e1a552656224c9facb4281c4d0de28d"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "c67669ba4ef7e92c6b2c10cb1df0bfaf"
server_key_iv: "2e7a9b693c497391393fbb1613ca902d"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.068411 8114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:42.069276 8129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.069562 8114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:42.069650 8114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "2e1a552656224c9facb4281c4d0de28d"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "c67669ba4ef7e92c6b2c10cb1df0bfaf"
server_key_iv: "2e7a9b693c497391393fbb1613ca902d"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.069720 8114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.087687 8114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:42.089753 8114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:42.089883 8114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:42.090122 8114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:42.090466 8114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:42.090546 8114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.090590 8114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:42.090617 8114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.096827 8114 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:42495
I20260620 22:56:42.096874 8242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:42495 every 8 connection(s)
I20260620 22:56:42.097203 8114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:42.102552 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 8114
I20260620 22:56:42.102700 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:56:42.102902 1688 external_mini_cluster.cc:1428] Setting key ec5c439064ddc30641063ae137da9585
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:56:42.111488 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.098049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:50519 (local address 127.1.166.62:33827)
0620 22:56:42.098132 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.098135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.098270 (+ 135us) server_negotiation.cc:407] Connection header received
0620 22:56:42.098883 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.098885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.098897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.098977 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.100013 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.100528 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.102013 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.102285 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.102428 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.108187 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.108189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.108189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.108198 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.109916 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.110442 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.110443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.110443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.110503 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.110862 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.110863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.110863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.110994 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.111056 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.111332 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.111416 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 22:56:42.111624 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.097985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:50519)
0620 22:56:42.098146 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.098190 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.098749 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.099045 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.099049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.099682 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.099902 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.099909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.100698 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.100700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.101829 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.101833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.102528 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.102531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.102801 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.106301 (+ 3500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.106314 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.108048 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.110023 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.110036 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.110038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.110329 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.110640 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.110641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.110641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.110762 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.111052 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.111056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.111193 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.111276 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.111423 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260620 22:56:42.112052 8243 heartbeater.cc:347] Connected to a master server at 127.1.166.62:33827
I20260620 22:56:42.112146 8243 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:42.112298 8243 heartbeater.cc:511] Master 127.1.166.62:33827 requested a full tablet report, sending...
I20260620 22:56:42.112668 7784 ts_manager.cc:194] Registered new tserver with Master: 2e1a552656224c9facb4281c4d0de28d (127.1.166.3:42495)
I20260620 22:56:42.113023 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:50519
I20260620 22:56:42.115391 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.166.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.61.keytab.
Entry for principal oryx/127.1.166.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.61.keytab.
I20260620 22:56:42.141377 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.61:37875
--webserver_interface=127.1.166.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.61.keytab
--principal=oryx/127.1.166.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:33827,127.1.166.61:37875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:42.233119 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:42.233472 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:42.233546 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:42.234863 8248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:42.234905 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:42.234925 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:42.234942 8248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:42.234958 8248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:42.236572 8248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:33827,127.1.166.61:37875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.61.keytab
--principal=oryx/127.1.166.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.61:37875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.166.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
W20260620 22:56:42.236690 8248 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.166.62:33827,127.1.166.61:37875'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 22:56:42.236850 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:42.237071 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:42.241748 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.241773 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.241926 8248 server_base.cc:1034] running on GCE node
W20260620 22:56:42.242004 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.242221 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:42.242524 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:42.243708 8248 hybrid_clock.cc:648] HybridClock initialized: now 1781996202243694 us; error 29 us; skew 500 ppm
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:42.246236 8248 init.cc:377] Logged in from keytab as oryx/127.1.166.61@KRBTEST.COM (short username oryx)
I20260620 22:56:42.246842 8248 webserver.cc:460] Webserver started at http://127.1.166.61:39691/ using document root <none> and password file <none>
I20260620 22:56:42.247061 8248 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.247108 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.247233 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:42.248351 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data/instance:
uuid: "64f56dd320604e1e9d7d50507c83681b"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "1407ab702b4afa04c84b4e188f567852"
server_key_iv: "e349f6e4db3bcb5b44d71b685fafe2f7"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.248849 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal/instance:
uuid: "64f56dd320604e1e9d7d50507c83681b"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "1407ab702b4afa04c84b4e188f567852"
server_key_iv: "e349f6e4db3bcb5b44d71b685fafe2f7"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.250555 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 22:56:42.251629 8263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.251832 8248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:42.251924 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
uuid: "64f56dd320604e1e9d7d50507c83681b"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "1407ab702b4afa04c84b4e188f567852"
server_key_iv: "e349f6e4db3bcb5b44d71b685fafe2f7"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.251997 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.262915 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:42.265434 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:42.265597 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:42.272090 8248 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.61:37875
I20260620 22:56:42.272153 8315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.61:37875 every 8 connection(s)
I20260620 22:56:42.272559 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
I20260620 22:56:42.273774 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:42.275535 8316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:42.277263 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 8248
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.61@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.166.61@KRBTEST.COM
I20260620 22:56:42.285902 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.276305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.1.166.62:33827)
0620 22:56:42.276378 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.276380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.276578 (+ 198us) server_negotiation.cc:407] Connection header received
0620 22:56:42.277114 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.277115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.277128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.277183 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.277951 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.278699 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.280052 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.280271 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.280367 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.282848 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.282850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.282850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.282858 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.284455 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.284881 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.284882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.284882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.284948 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.285229 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.285229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.285230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.285393 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.285449 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.285747 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.285831 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:56:42.286096 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.276262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46026)
0620 22:56:42.276457 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.276498 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.277013 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.277259 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.277263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.277579 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.277864 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.277872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.278792 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.278793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.279950 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.279953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.280386 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.280387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.280492 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.281043 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.281053 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.282756 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.284539 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.284547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.284550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.284818 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.285008 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.285009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.285009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.285139 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.285453 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.285459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.285620 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.285703 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.285843 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260620 22:56:42.286702 8316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:42.286809 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.277935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.1.166.61:37875)
0620 22:56:42.278103 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.278106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.278422 (+ 316us) server_negotiation.cc:407] Connection header received
0620 22:56:42.278455 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.278457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.278478 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.278544 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.279168 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.280147 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.280986 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.281400 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.281481 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.283979 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.283987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.283987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.283993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.285461 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.285803 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.285805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.285806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.285862 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.286067 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.286068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.286069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.286246 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.286424 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.286609 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.286690 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.61@KRBTEST.COM for oryx/127.1.166.61@KRBTEST.COM
I20260620 22:56:42.294536 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.286961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.0.0.1:33768)
0620 22:56:42.287000 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.287010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.287045 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.287161 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.287161 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.287290 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.287326 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.287328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.288027 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.288028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.288777 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.288779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.289340 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.289341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.289397 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.289831 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.289835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.291518 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.293543 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.293544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.293545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.293722 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.293898 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.293898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.293899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.293945 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.294158 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.294159 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.294198 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.294255 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.294411 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 22:56:42.294543 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.287045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.1.166.61:37875)
0620 22:56:42.287077 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.287078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.287085 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:42.287103 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.287103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.287113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.287155 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.287426 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.287920 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.288959 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.289204 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.289341 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.291859 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.291861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.291861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.291872 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.293433 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.293793 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.293794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.293794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.293843 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.293994 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.293995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.293995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.294104 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.294158 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.294279 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.294462 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":16}
I20260620 22:56:42.296986 8316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Bootstrap starting.
I20260620 22:56:42.298054 8316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:42.298416 8316 log.cc:824] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:42.299490 8316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: No bootstrap required, opened a new log
I20260620 22:56:42.301131 8316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } }
I20260620 22:56:42.301301 8316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:42.301323 8316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f56dd320604e1e9d7d50507c83681b, State: Initialized, Role: FOLLOWER
I20260620 22:56:42.301438 8316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } }
I20260620 22:56:42.301803 8316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:42.301795 8322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } } }
I20260620 22:56:42.301919 8322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:56:42.303618 8333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:42.303690 8333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:42.306315 8248 master_runner.cc:428] Detected that this master 64f56dd320604e1e9d7d50507c83681b is joining an existing cluster
I20260620 22:56:42.306448 8248 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.166.61:37875
I20260620 22:56:42.316294 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.307854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.0.0.1:33776)
0620 22:56:42.308080 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.308091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.308189 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.308342 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.308342 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.308852 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.308957 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.308960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.310032 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.310033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.310966 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.310969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.311524 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.311525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.311602 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.312374 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.312379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.312815 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.314741 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.314742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.314743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.315160 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.315384 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.315385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.315385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.315460 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.315874 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.315874 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.315934 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.315999 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.316157 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260620 22:56:42.316303 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.307873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.1.166.61:37875)
0620 22:56:42.307924 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.307926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.308162 (+ 236us) server_negotiation.cc:407] Connection header received
0620 22:56:42.308248 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.308249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.308257 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.308284 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.309067 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.309939 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.311082 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.311391 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.311502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.312946 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.312947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.312947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.312956 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.314632 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.315260 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.315262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.315262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.315315 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.315640 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.315643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.315644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.315815 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.315884 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.316033 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.316226 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":27}
I20260620 22:56:42.318097 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.307956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.1.166.62:33827)
0620 22:56:42.308010 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.308012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.308204 (+ 192us) server_negotiation.cc:407] Connection header received
0620 22:56:42.308294 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.308295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.308308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.308354 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.309050 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.309941 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.311656 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.312038 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.312125 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.313294 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.313295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.313296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.313305 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.314875 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.315386 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.315387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.315388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.315443 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.315817 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.315819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.315820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.316011 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.316088 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.317906 (+ 1818us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.318021 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:56:42.318149 8340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.307948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46036)
0620 22:56:42.308078 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.308088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.308194 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.308364 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.308364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.308834 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.308940 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.308943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.310638 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.310640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.311472 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.311475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.312125 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.312126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.312205 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.312700 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.312705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.313190 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.314941 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.314942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.314942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.315192 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.315501 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.315501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.315502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.315694 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.316117 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.316118 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.317718 (+ 1600us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.317847 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.318004 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":14,"thread_start_us":59,"threads_started":1}
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.62@KRBTEST.COM for oryx/127.1.166.61@KRBTEST.COM
I20260620 22:56:42.328725 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.321513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.1.166.61:37875)
0620 22:56:42.321587 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.321589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.321598 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:42.321656 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.321656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.321667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.321714 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.322167 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.322700 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.323456 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.323661 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.323716 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.326217 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.326219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.326220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.326229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.327585 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.327998 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.328000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.328000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.328056 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.328221 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.328222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.328223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.328329 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.328384 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.328549 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.328623 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:56:42.328750 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.321295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.0.0.1:33788)
0620 22:56:42.321469 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.321506 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.321576 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.321724 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.321728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.322032 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.322092 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.322094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.322770 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.322771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.323378 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.323380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.323721 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.323722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.323807 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.324318 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.324322 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.326034 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.327685 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.327695 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.327698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.327931 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.328112 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.328112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.328113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.328173 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.328383 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.328391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.328446 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.328518 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.328647 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260620 22:56:42.329604 7784 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.166.61:37875: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "64f56dd320604e1e9d7d50507c83681b"
member_type: NON_VOTER
last_known_addr {
host: "127.1.166.61"
port: 37875
}
attrs {
promote: true
}
}
dest_uuid: "90020d0371ed4eaf9ef69a85170700a2"
cas_config_opid_index: -1
I20260620 22:56:42.329919 7784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } }
I20260620 22:56:42.330400 8343 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 64f56dd320604e1e9d7d50507c83681b (127.1.166.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } }
I20260620 22:56:42.330734 8290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:42.331284 8343 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.166.61:37875
I20260620 22:56:42.331380 8343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } }
I20260620 22:56:42.331444 8343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:42.331542 8248 master.cc:537] Master@127.1.166.61:37875 shutting down...
I20260620 22:56:42.331739 8290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Refusing update from remote peer 90020d0371ed4eaf9ef69a85170700a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 22:56:42.332155 8342 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:56:42.333050 7770 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 22:56:42.333988 8248 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:42.334085 8248 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:42.334137 8248 tablet_replica.cc:330] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: stopping tablet replica
I20260620 22:56:42.338836 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.338705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.0.0.1:33806)
0620 22:56:42.338755 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.338794 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:37875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:56:42.338982 7770 consensus_peers.cc:487] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 -> Peer 64f56dd320604e1e9d7d50507c83681b (127.1.166.61:37875): Couldn't send request to peer 64f56dd320604e1e9d7d50507c83681b. Status: Network error: Client connection negotiation failed: client connection to 127.1.166.61:37875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 22:56:42.348100 8248 master.cc:552] Master@127.1.166.61:37875 shutdown complete.
I20260620 22:56:42.348176 8248 master_runner.cc:305] Clearing existing system tablet
I20260620 22:56:42.348332 8248 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.348388 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.349445 8248 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:42.350178 8347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.350463 8248 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.350553 8248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:56:42.350581 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
uuid: "64f56dd320604e1e9d7d50507c83681b"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "1407ab702b4afa04c84b4e188f567852"
server_key_iv: "e349f6e4db3bcb5b44d71b685fafe2f7"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.351656 8248 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:42.353717 8248 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 22:56:42.353818 8248 log.cc:1196] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 22:56:42.354074 8248 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Deleting consensus metadata
I20260620 22:56:42.354206 8248 master_runner.cc:315] Copying system tablet from 127.1.166.62:33827
I20260620 22:56:42.355011 8248 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: tablet copy: Beginning tablet copy session from remote peer at address 127.1.166.62:33827
I20260620 22:56:42.363663 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.355488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46038)
0620 22:56:42.355671 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.355684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.355798 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.356176 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.356178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.356383 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.356460 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.356470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.357611 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.357612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.358226 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.358228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.358770 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.358771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.358831 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.359878 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.359884 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.360293 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.362103 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.362104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.362104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.362308 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.362624 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.362625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.362625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.362689 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.363288 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.363289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.363350 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.363434 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.363560 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
I20260620 22:56:42.363801 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.355763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.1.166.62:33827)
0620 22:56:42.355884 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.355887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.355903 (+ 16us) server_negotiation.cc:407] Connection header received
0620 22:56:42.355947 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.355948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.355964 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.356059 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.356637 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.357452 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.358319 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.358677 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.358759 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.360393 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.360395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.360396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.360405 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.361980 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.362449 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.362450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.362450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.362518 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.362840 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.362843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.362844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.363128 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.363296 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.363574 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.363696 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 22:56:42.364383 7804 tablet_copy_service.cc:140] P 90020d0371ed4eaf9ef69a85170700a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 64f56dd320604e1e9d7d50507c83681b ({username='oryx', principal='oryx/127.1.166.61@KRBTEST.COM'} at 127.0.0.1:46038)
I20260620 22:56:42.364490 7804 tablet_copy_service.cc:161] P 90020d0371ed4eaf9ef69a85170700a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 64f56dd320604e1e9d7d50507c83681b at {username='oryx', principal='oryx/127.1.166.61@KRBTEST.COM'} at 127.0.0.1:46038: session id = 64f56dd320604e1e9d7d50507c83681b-00000000000000000000000000000000
I20260620 22:56:42.365191 7804 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:56:42.366153 8248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 22:56:42.368055 8248 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: tablet copy: Starting download of 0 data blocks...
I20260620 22:56:42.368254 8248 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: tablet copy: Starting download of 1 WAL segments...
I20260620 22:56:42.369537 8248 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:56:42.371490 7804 tablet_copy_service.cc:342] P 90020d0371ed4eaf9ef69a85170700a2: Request end of tablet copy session 64f56dd320604e1e9d7d50507c83681b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.166.61@KRBTEST.COM'} at 127.0.0.1:46038
I20260620 22:56:42.371618 7804 tablet_copy_service.cc:433] P 90020d0371ed4eaf9ef69a85170700a2: ending tablet copy session 64f56dd320604e1e9d7d50507c83681b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 64f56dd320604e1e9d7d50507c83681b
I20260620 22:56:42.375712 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:42.377060 8355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.377060 8354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.377511 8357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.377605 8248 server_base.cc:1034] running on GCE node
I20260620 22:56:42.377782 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:42.377950 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:42.379086 8248 hybrid_clock.cc:648] HybridClock initialized: now 1781996202379072 us; error 33 us; skew 500 ppm
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:42.381098 8248 init.cc:377] Logged in from keytab as oryx/127.1.166.61@KRBTEST.COM (short username oryx)
I20260620 22:56:42.381386 8248 webserver.cc:460] Webserver started at http://127.1.166.61:38247/ using document root <none> and password file <none>
I20260620 22:56:42.381510 8248 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.381541 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.382144 8248 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:42.382723 8364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.382905 8248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.382977 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
uuid: "64f56dd320604e1e9d7d50507c83681b"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "1407ab702b4afa04c84b4e188f567852"
server_key_iv: "e349f6e4db3bcb5b44d71b685fafe2f7"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.383041 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.401808 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:42.403589 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:42.406440 8248 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.61:37875
I20260620 22:56:42.406492 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.61:37875 every 8 connection(s)
I20260620 22:56:42.406970 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
I20260620 22:56:42.407904 8417 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:56:42.408499 8417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Bootstrap starting.
I20260620 22:56:42.411717 8417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:56:42.411976 8417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: Bootstrap complete.
I20260620 22:56:42.412230 8417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } }
I20260620 22:56:42.412343 8417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64f56dd320604e1e9d7d50507c83681b, State: Initialized, Role: LEARNER
I20260620 22:56:42.412385 8417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } }
I20260620 22:56:42.412627 8417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:42.412683 8420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } }
I20260620 22:56:42.412876 8420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: This master's current role is: LEARNER
I20260620 22:56:42.414916 8431 catalog_manager.cc:1259] Loaded cluster ID: e6d30f27358c40418885647fcbc0c04b
I20260620 22:56:42.414979 8431 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: loading cluster ID for follower catalog manager: success
I20260620 22:56:42.415799 8431 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: acquiring CA information for follower catalog manager: success
I20260620 22:56:42.416097 8431 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:56:42.447304 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.439275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.1.166.61:37875)
0620 22:56:42.439499 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.439501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.439511 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:42.439538 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.439539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.439550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.439620 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.440396 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.441106 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.442128 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.442374 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.442440 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.444159 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.444161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.444162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.444170 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.445628 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.446101 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.446102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.446103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.446171 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.446490 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.446492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.446492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.446739 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.446873 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.447047 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.447157 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for oryx/127.1.166.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.60.keytab.
Entry for principal oryx/127.1.166.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.60.keytab.
I20260620 22:56:42.475605 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.60:38715
--webserver_interface=127.1.166.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.60.keytab
--principal=oryx/127.1.166.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:42.564960 8435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:42.565189 8435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:42.565213 8435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:42.566666 8435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:42.566736 8435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:42.566751 8435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:42.566762 8435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:42.566772 8435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:42.568538 8435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.60.keytab
--principal=oryx/127.1.166.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.60:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.166.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:42.568821 8435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:42.569099 8435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:42.574640 8443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.574782 8435 server_base.cc:1034] running on GCE node
W20260620 22:56:42.574656 8440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.574900 8441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.575227 8435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:42.575608 8435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:42.576788 8435 hybrid_clock.cc:648] HybridClock initialized: now 1781996202576750 us; error 65 us; skew 500 ppm
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:42.579627 8435 init.cc:377] Logged in from keytab as oryx/127.1.166.60@KRBTEST.COM (short username oryx)
I20260620 22:56:42.580161 8435 webserver.cc:460] Webserver started at http://127.1.166.60:43489/ using document root <none> and password file <none>
I20260620 22:56:42.580382 8435 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.580440 8435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.580567 8435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:42.581589 8435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data/instance:
uuid: "f5e6eebf0fea49aa91a389a3331f88a9"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "2b628a5ce32d7974c155b421b2c3ece1"
server_key_iv: "ef0d36fafa736f04f1f4ce32819e12b6"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.581954 8435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal/instance:
uuid: "f5e6eebf0fea49aa91a389a3331f88a9"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "2b628a5ce32d7974c155b421b2c3ece1"
server_key_iv: "ef0d36fafa736f04f1f4ce32819e12b6"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.583503 8435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:42.584303 8450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.584448 8435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:56:42.584539 8435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
uuid: "f5e6eebf0fea49aa91a389a3331f88a9"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "2b628a5ce32d7974c155b421b2c3ece1"
server_key_iv: "ef0d36fafa736f04f1f4ce32819e12b6"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.584614 8435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.605458 8435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:42.607955 8435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:42.608121 8435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:42.614728 8435 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.60:38715
I20260620 22:56:42.614795 8502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.60:38715 every 8 connection(s)
I20260620 22:56:42.615242 8435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
I20260620 22:56:42.616204 8503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:42.617888 8503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:42.622411 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 8435
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.60@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.166.60@KRBTEST.COM
I20260620 22:56:42.629073 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.618948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.1.166.62:33827)
0620 22:56:42.619018 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.619021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.619038 (+ 17us) server_negotiation.cc:407] Connection header received
0620 22:56:42.619744 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.619745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.619759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.619829 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.620524 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.621067 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.622288 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.622676 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.622808 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.625355 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.625357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.625357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.625369 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.627289 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.627846 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.627847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.627847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.627919 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.628251 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.628253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.628254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.628407 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.628493 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.628877 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.628978 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:56:42.629246 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.618713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46050)
0620 22:56:42.618924 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.618969 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.619584 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.619916 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.619919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.620199 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.620419 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.620423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.621204 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.621205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.622121 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.622125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.622859 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.622861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.623106 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.623673 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.623683 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.625248 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.627453 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.627460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.627462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.627749 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.628024 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.628024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.628025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.628144 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.628486 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.628493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.628657 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.628804 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.628990 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260620 22:56:42.630129 8503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:42.632489 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.623164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.1.166.60:38715)
0620 22:56:42.623401 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.623405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.623420 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:56:42.623440 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.623442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.623461 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.623527 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.624189 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.624932 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.625829 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.626214 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.626280 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.629218 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.629226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.629226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.629234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.631066 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.631516 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.631520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.631521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.631653 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.631887 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.631887 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.631888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.632095 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.632234 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.632318 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.632389 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":145,"threads_started":1}
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.60@KRBTEST.COM for oryx/127.1.166.61@KRBTEST.COM
I20260620 22:56:42.639070 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.630495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.0.0.1:33840)
0620 22:56:42.630554 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.630567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.630613 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.630885 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.630895 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.631087 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.631159 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.631161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.632106 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.632107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.633164 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.633167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.633750 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.633752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.633857 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.634594 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.634599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.636140 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.637923 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.637925 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.637925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.638136 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.638390 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.638391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.638392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.638451 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.638783 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.638784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.638833 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.638889 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.638991 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:56:42.639092 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.630615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.1.166.61:37875)
0620 22:56:42.630697 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.630700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.630711 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:42.630742 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.630743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.630759 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.630838 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.631276 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.632014 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.633313 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.633635 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.633708 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.636293 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.636295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.636296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.636306 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.637765 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.638246 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.638247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.638248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.638302 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.638550 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.638551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.638551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.638701 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.638786 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.638925 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.639020 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 22:56:42.639549 8503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 38715 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.60@KRBTEST.COM for oryx/127.1.166.60@KRBTEST.COM
I20260620 22:56:42.647927 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.639859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:38715 (local address 127.0.0.1:46976)
0620 22:56:42.639903 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.639915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.639958 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.640253 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.640254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.640433 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.640490 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.640492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.641264 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.641266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.641776 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.641777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.642220 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.642221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.642271 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.642772 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.642776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.644410 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.646675 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.646677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.646677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.646958 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.647239 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.647240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.647240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.647311 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.647627 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.647628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.647686 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.647755 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.647849 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 22:56:42.648102 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.640004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.1.166.60:38715)
0620 22:56:42.640072 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.640074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.640083 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:42.640111 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.640112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.640125 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.640196 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.640627 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.641139 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.641890 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.642112 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.642181 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.644574 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.644576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.644576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.644587 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.646499 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.647075 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.647076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.647077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.647147 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.647378 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.647378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.647379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.647502 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.647635 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.647867 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.647992 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:56:42.649772 8503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Bootstrap starting.
I20260620 22:56:42.650660 8503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:42.650887 8503 log.cc:824] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:42.651647 8503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: No bootstrap required, opened a new log
I20260620 22:56:42.653069 8503 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 38715 } }
I20260620 22:56:42.653199 8503 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:42.653221 8503 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e6eebf0fea49aa91a389a3331f88a9, State: Initialized, Role: FOLLOWER
I20260620 22:56:42.653330 8503 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 38715 } }
I20260620 22:56:42.653715 8508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 38715 } } }
I20260620 22:56:42.653811 8508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:42.653831 8503 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 22:56:42.656069 8519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:42.656134 8519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:42.659958 8435 master_runner.cc:428] Detected that this master f5e6eebf0fea49aa91a389a3331f88a9 is joining an existing cluster
I20260620 22:56:42.660041 8435 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.166.60:38715
I20260620 22:56:42.669622 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.661310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.1.166.61:37875)
0620 22:56:42.661363 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.661365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.661439 (+ 74us) server_negotiation.cc:407] Connection header received
0620 22:56:42.661508 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.661509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.661521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.661572 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.662313 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.663155 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.664300 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.664601 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.664664 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.666488 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.666489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.666490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.666497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.667821 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.668450 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.668452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.668453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.668537 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.668826 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.668828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.668828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.669018 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.669097 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.669460 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.669548 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:56:42.669706 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.661185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.0.0.1:33850)
0620 22:56:42.661370 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.661381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.661447 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.661620 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.661621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.662060 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.662200 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.662203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.663267 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.663268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.664195 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.664198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.665045 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.665046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.665136 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.665721 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.665726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.666224 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.667939 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.667941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.667942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.668289 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.668618 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.668619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.668620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.668703 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.669102 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.669103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.669300 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.669407 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.669570 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260620 22:56:42.684664 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7846
I20260620 22:56:42.691144 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.661772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.1.166.60:38715)
0620 22:56:42.661807 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.661809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.662480 (+ 671us) server_negotiation.cc:407] Connection header received
0620 22:56:42.662558 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.662559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.662568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.662592 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.663329 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.664011 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.665064 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.665331 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.665371 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.688304 (+ 22933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.688306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.688306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.688315 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.689926 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.690315 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.690316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.690316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.690428 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.690619 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.690620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.690621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.690760 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.690802 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.690968 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.691050 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:56:42.691480 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.661839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:38715 (local address 127.0.0.1:46992)
0620 22:56:42.662409 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.662425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.662501 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.662831 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.662832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.663138 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.663212 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.663214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.664093 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.664094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.664937 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.664942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.665441 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.665442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.665505 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.687319 (+ 21814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.687327 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.688088 (+ 761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.690032 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.690033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.690034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.690239 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.690484 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.690484 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.690485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.690551 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.690840 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.690841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.690878 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.691267 (+ 389us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.691369 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":65,"threads_started":1}
I20260620 22:56:42.691521 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:38073
--local_ip_for_outbound_sockets=127.1.166.1
--tserver_master_addrs=127.1.166.62:33827
--webserver_port=44315
--webserver_interface=127.1.166.1
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab
--principal=oryx/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:56:42.691679 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.661506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.1.166.62:33827)
0620 22:56:42.661558 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.661560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.661902 (+ 342us) server_negotiation.cc:407] Connection header received
0620 22:56:42.662063 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.662064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.662083 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.662114 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.662717 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.663455 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.664479 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.664918 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.665005 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.688088 (+ 23083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.688091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.688091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.688102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.690184 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.690732 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.690733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.690734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.690791 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.690968 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.690969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.690969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.691093 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.691141 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.691507 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.691593 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:56:42.691977 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.661583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46064)
0620 22:56:42.661708 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.661717 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.661968 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.662219 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.662219 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.662472 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.662552 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.662553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.663558 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.663559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.664389 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.664391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.665143 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.665144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.665226 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.687185 (+ 21959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.687194 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.687931 (+ 737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.690297 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.690298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.690299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.690649 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.690857 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.690858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.690858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.690910 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.691176 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.691177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.691299 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.691772 (+ 473us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.691878 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996201, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.62@KRBTEST.COM for oryx/127.1.166.60@KRBTEST.COM
I20260620 22:56:42.704736 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.695656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.1.166.60:38715)
0620 22:56:42.695715 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.695717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.695726 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:42.695771 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.695772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.695784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.695852 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.696203 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.696978 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.698091 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.698461 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.698552 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.701350 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.701351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.701352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.701363 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.703100 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.703745 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.703746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.703746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.703819 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.704125 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.704125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.704126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.704268 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.704329 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.704549 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.704633 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:56:42.704782 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.695447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:38715 (local address 127.0.0.1:47008)
0620 22:56:42.695508 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.695537 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.695577 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.695871 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.695872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.696064 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.696123 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.696126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.697075 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.697076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.697991 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.697993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.698549 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.698551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.698626 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.699513 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.699518 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.701211 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.703278 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.703280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.703281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.703623 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.703915 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.703916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.703917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.704043 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.704357 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.704358 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.704421 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.704520 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.704675 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 22:56:42.705821 7784 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.166.60:38715: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9"
member_type: NON_VOTER
last_known_addr {
host: "127.1.166.60"
port: 38715
}
attrs {
promote: true
}
}
dest_uuid: "90020d0371ed4eaf9ef69a85170700a2"
cas_config_opid_index: 5
I20260620 22:56:42.706085 7784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:42.706524 8342 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f5e6eebf0fea49aa91a389a3331f88a9 (127.1.166.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.706753 8477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:42.707499 8477 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 1 FOLLOWER]: Refusing update from remote peer 90020d0371ed4eaf9ef69a85170700a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 22:56:42.707520 8342 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.166.60:38715
I20260620 22:56:42.707582 8342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.707647 8342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:42.707769 8343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:42.708236 8435 master.cc:537] Master@127.1.166.60:38715 shutting down...
I20260620 22:56:42.709060 8477 raft_consensus.cc:858] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 38715 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
W20260620 22:56:42.709121 8477 raft_consensus.cc:1565] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 38715 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:42.709764 8477 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 64f56dd320604e1e9d7d50507c83681b (127.1.166.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } }
I20260620 22:56:42.710860 8535 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } }
I20260620 22:56:42.711011 8535 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 22:56:42.710951 8508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } }
I20260620 22:56:42.711475 8508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 22:56:42.711890 8529 mvcc.cc:205] Tried to move back new op lower bound from 7299056441791602688 to 7299056441714872320. Current Snapshot: MvccSnapshot[applied={T|T < 7299056441791602688 or (T in {7299056441791602688})}]
I20260620 22:56:42.712080 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.708105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.0.0.1:33852)
0620 22:56:42.708326 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.708337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.708394 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.708642 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.708643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.708645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:42.708695 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.708696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.709562 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.709564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.710939 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.710942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.711782 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.711783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.711861 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.711886 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.711948 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.712007 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260620 22:56:42.712558 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.708252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.1.166.61:37875)
0620 22:56:42.708296 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.708298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.708371 (+ 73us) server_negotiation.cc:407] Connection header received
0620 22:56:42.708476 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.708477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.708505 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.708546 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:42.708769 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.709484 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.711046 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.711673 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.711751 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.711818 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.712430 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.712480 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:56:42.712862 8391 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 LEARNER]: Refusing update from remote peer 90020d0371ed4eaf9ef69a85170700a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 22:56:42.712812 8343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.712898 8343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:42.713166 8342 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 22:56:42.713367 8531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90020d0371ed4eaf9ef69a85170700a2. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } }
I20260620 22:56:42.713464 8531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 22:56:42.713791 8391 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f5e6eebf0fea49aa91a389a3331f88a9 (127.1.166.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.715142 8342 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: attempting to promote NON_VOTER 64f56dd320604e1e9d7d50507c83681b to VOTER
I20260620 22:56:42.715137 8420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90020d0371ed4eaf9ef69a85170700a2. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.715229 8420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: This master's current role is: LEARNER
I20260620 22:56:42.715263 8420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: NON_VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: true } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.715301 8420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: This master's current role is: LEARNER
I20260620 22:56:42.715318 8342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:42.715684 8435 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 22:56:42.715792 8435 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 22:56:42.715821 8435 tablet_replica.cc:330] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: stopping tablet replica
W20260620 22:56:42.716240 7769 consensus_peers.cc:487] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 -> Peer f5e6eebf0fea49aa91a389a3331f88a9 (127.1.166.60:38715): Couldn't send request to peer f5e6eebf0fea49aa91a389a3331f88a9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:56:42.716339 8391 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 LEARNER]: Refusing update from remote peer 90020d0371ed4eaf9ef69a85170700a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 22:56:42.716513 8343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 22:56:42.717154 8343 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 64f56dd320604e1e9d7d50507c83681b (127.1.166.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.718274 8343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.718413 8343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:42.718431 8538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.718525 8538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:42.719002 8391 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 64f56dd320604e1e9d7d50507c83681b (127.1.166.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.720299 8537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90020d0371ed4eaf9ef69a85170700a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.720523 8537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:42.732326 8435 master.cc:552] Master@127.1.166.60:38715 shutdown complete.
I20260620 22:56:42.732435 8435 master_runner.cc:305] Clearing existing system tablet
I20260620 22:56:42.732738 8435 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.732824 8435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.733742 8435 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:42.734642 8540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.734905 8435 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.735005 8435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:56:42.735039 8435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
uuid: "f5e6eebf0fea49aa91a389a3331f88a9"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "2b628a5ce32d7974c155b421b2c3ece1"
server_key_iv: "ef0d36fafa736f04f1f4ce32819e12b6"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.736107 8435 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:42.737847 8435 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 22:56:42.737957 8435 log.cc:1196] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 22:56:42.738209 8435 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Deleting consensus metadata
I20260620 22:56:42.738382 8435 master_runner.cc:315] Copying system tablet from 127.1.166.62:33827
I20260620 22:56:42.739105 8435 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: tablet copy: Beginning tablet copy session from remote peer at address 127.1.166.62:33827
I20260620 22:56:42.747560 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.739540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46080)
0620 22:56:42.739708 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.739720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.739793 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.739982 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.739983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.740211 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.740278 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.740280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.741325 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.741326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.742261 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.742264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.742984 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.742985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.743054 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.743630 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.743636 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.744079 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.746158 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.746159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.746160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.746442 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.746747 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.746748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.746749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.746823 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.747150 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.747151 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.747209 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.747286 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.747417 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260620 22:56:42.748131 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.739620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.1.166.62:33827)
0620 22:56:42.739682 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.739684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.739790 (+ 106us) server_negotiation.cc:407] Connection header received
0620 22:56:42.739859 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.739860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.739872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.739962 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.740370 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.741216 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.742419 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.742755 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.742851 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.744189 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.744191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.744191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.744200 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.745979 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.746555 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.746557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.746558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.746666 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.746895 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.746896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.746896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.747068 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.747149 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.747334 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.748033 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:56:42.748680 7804 tablet_copy_service.cc:140] P 90020d0371ed4eaf9ef69a85170700a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f5e6eebf0fea49aa91a389a3331f88a9 ({username='oryx', principal='oryx/127.1.166.60@KRBTEST.COM'} at 127.0.0.1:46080)
I20260620 22:56:42.748765 7804 tablet_copy_service.cc:161] P 90020d0371ed4eaf9ef69a85170700a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f5e6eebf0fea49aa91a389a3331f88a9 at {username='oryx', principal='oryx/127.1.166.60@KRBTEST.COM'} at 127.0.0.1:46080: session id = f5e6eebf0fea49aa91a389a3331f88a9-00000000000000000000000000000000
I20260620 22:56:42.749444 7804 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:56:42.750020 8435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 22:56:42.751466 8435 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: tablet copy: Starting download of 0 data blocks...
I20260620 22:56:42.751626 8435 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: tablet copy: Starting download of 1 WAL segments...
I20260620 22:56:42.752451 8435 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:56:42.753474 7804 tablet_copy_service.cc:342] P 90020d0371ed4eaf9ef69a85170700a2: Request end of tablet copy session f5e6eebf0fea49aa91a389a3331f88a9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.166.60@KRBTEST.COM'} at 127.0.0.1:46080
I20260620 22:56:42.753547 7804 tablet_copy_service.cc:433] P 90020d0371ed4eaf9ef69a85170700a2: ending tablet copy session f5e6eebf0fea49aa91a389a3331f88a9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f5e6eebf0fea49aa91a389a3331f88a9
I20260620 22:56:42.756960 8435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:42.758324 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.758458 8435 server_base.cc:1034] running on GCE node
W20260620 22:56:42.758654 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.763351 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.763711 8435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:42.763902 8435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:42.765055 8435 hybrid_clock.cc:648] HybridClock initialized: now 1781996202765048 us; error 29 us; skew 500 ppm
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:42.766708 8435 init.cc:377] Logged in from keytab as oryx/127.1.166.60@KRBTEST.COM (short username oryx)
I20260620 22:56:42.767040 8435 webserver.cc:460] Webserver started at http://127.1.166.60:40983/ using document root <none> and password file <none>
I20260620 22:56:42.767160 8435 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.767195 8435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.767974 8435 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:42.768568 8557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.768748 8435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.768805 8435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
uuid: "f5e6eebf0fea49aa91a389a3331f88a9"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "2b628a5ce32d7974c155b421b2c3ece1"
server_key_iv: "ef0d36fafa736f04f1f4ce32819e12b6"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.768877 8435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.785307 8435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:42.787534 8435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:42.791157 8435 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.60:38715
I20260620 22:56:42.791556 8435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
I20260620 22:56:42.792555 8610 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:56:42.793138 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Bootstrap starting.
I20260620 22:56:42.794516 8609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.60:38715 every 8 connection(s)
I20260620 22:56:42.795922 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:56:42.796164 8610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: Bootstrap complete.
I20260620 22:56:42.796388 8610 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:42.796486 8610 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5e6eebf0fea49aa91a389a3331f88a9, State: Initialized, Role: LEARNER
I20260620 22:56:42.796526 8610 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:42.796748 8610 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:42.796761 8613 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } } }
I20260620 22:56:42.797053 8613 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9 [sys.catalog]: This master's current role is: LEARNER
I20260620 22:56:42.798424 8624 catalog_manager.cc:1259] Loaded cluster ID: e6d30f27358c40418885647fcbc0c04b
I20260620 22:56:42.798640 8624 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: loading cluster ID for follower catalog manager: success
I20260620 22:56:42.799906 8624 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: acquiring CA information for follower catalog manager: success
I20260620 22:56:42.800326 8624 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5e6eebf0fea49aa91a389a3331f88a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 22:56:42.810808 8528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:42.811072 8528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:42.811105 8528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:42.812515 8528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:42.812633 8528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:42.812749 8528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:56:42.815354 8528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab
--principal=oryx/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:38073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=44315
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:42.815739 8528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:42.816025 8528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:42.821556 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.821624 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.821971 8528 server_base.cc:1034] running on GCE node
W20260620 22:56:42.821624 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.822269 8528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:42.822602 8528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:42.823812 8528 hybrid_clock.cc:648] HybridClock initialized: now 1781996202823796 us; error 31 us; skew 500 ppm
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:42.826879 8528 init.cc:377] Logged in from keytab as oryx/127.1.166.1@KRBTEST.COM (short username oryx)
I20260620 22:56:42.827454 8528 webserver.cc:460] Webserver started at http://127.1.166.1:44315/ using document root <none> and password file <none>
I20260620 22:56:42.827895 8528 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.828004 8528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.829501 8528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:42.830124 8639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.830292 8528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:56:42.830400 8528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "550e61ba7c3542bb9b06f7c1baaf87a2"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "dd10680d420e797360e0927fe87aaa5d"
server_key_iv: "9acb0fd2cbbee8589ca538e8fb79a077"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.830765 8528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.839879 8528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:42.842788 8528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:42.842973 8528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:42.843216 8528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:42.843536 8528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:42.843569 8528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.843600 8528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:42.843626 8528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.849874 8528 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:38073
I20260620 22:56:42.850006 8752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:38073 every 8 connection(s)
I20260620 22:56:42.850343 8528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.60@KRBTEST.COM
I20260620 22:56:42.858462 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 8528
I20260620 22:56:42.858624 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7980
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.61@KRBTEST.COM
I20260620 22:56:42.861953 8757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.851117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:52165)
0620 22:56:42.851301 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.851344 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.852468 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.852847 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.852848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.853147 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.853395 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.853400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.854400 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.854401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.855349 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.855351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.855789 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.855867 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.856520 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.856531 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.858295 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.860230 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.860240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.860243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.860551 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.860741 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.860742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.860742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.860891 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.861186 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.861193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.861380 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.861490 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.861678 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":28,"thread_start_us":86,"threads_started":1}
I20260620 22:56:42.862259 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.851105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:52165 (local address 127.1.166.62:33827)
0620 22:56:42.851157 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.851159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.851429 (+ 270us) server_negotiation.cc:407] Connection header received
0620 22:56:42.852719 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.852720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.852733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.852791 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.853489 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.854283 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855436 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.855722 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855775 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.858442 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.858444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.858444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.858453 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.860155 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.860617 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.860618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.860619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.860673 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.860963 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.860964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.860964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.861096 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.861153 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.862092 (+ 939us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.862171 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:42.862957 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.851424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:38715 (local address 127.1.166.1:35195)
0620 22:56:42.851538 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.851546 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.852453 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.852795 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.852795 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.853167 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.853399 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.853401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.854612 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.854612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.855231 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.855234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.855698 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.855844 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.856635 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.856640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.859506 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.861821 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.861821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.861822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.862004 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.862195 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.862196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.862196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.862372 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.862604 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.862605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.862648 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.862718 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.862821 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":3,"thread_start_us":56,"threads_started":1}
I20260620 22:56:42.863095 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.851509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:35195 (local address 127.1.166.60:38715)
0620 22:56:42.851619 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.851621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.851629 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:42.852655 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.852656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.852678 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.852775 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.853513 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.854540 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855315 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.855613 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855682 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.860218 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.860221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.860222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.860231 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.861732 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.862078 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.862079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.862079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.862133 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.862427 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.862428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.862428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.862546 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.862604 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.862918 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.863002 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260620 22:56:42.864344 8754 heartbeater.cc:347] Connected to a master server at 127.1.166.62:33827
I20260620 22:56:42.864456 8754 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:42.864653 8754 heartbeater.cc:511] Master 127.1.166.62:33827 requested a full tablet report, sending...
I20260620 22:56:42.865077 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.851115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:34993 (local address 127.1.166.61:37875)
0620 22:56:42.851165 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:42.851167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:42.851423 (+ 256us) server_negotiation.cc:407] Connection header received
0620 22:56:42.852568 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:42.852570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:42.852583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:42.852612 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:42.853502 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.854022 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855464 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.855702 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855765 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.859920 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:42.859922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:42.859922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:42.859931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:42.861880 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.862724 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.862726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.862726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.862786 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:42.864472 (+ 1686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:42.864474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:42.864474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:42.864640 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:42.864695 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:42.864903 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 22:56:42.864988 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:42.865079 7784 ts_manager.cc:194] Re-registered known tserver with Master: 550e61ba7c3542bb9b06f7c1baaf87a2 (127.1.166.1:38073)
I20260620 22:56:42.865417 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:42.851117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.1.166.1:34993)
0620 22:56:42.851308 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:42.851344 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:42.852439 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:42.852732 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:42.852736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:42.853158 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:42.853397 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.853401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.854122 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.854122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.855362 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:42.855363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:42.855821 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:42.855822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:42.855877 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:42.856630 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:42.856640 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:42.859801 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:42.861974 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.861975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.861976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.862648 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.862858 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:42.862859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:42.862859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:42.862939 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:42.864732 (+ 1793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:42.864732 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:42.864788 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 22:56:42.865194 (+ 406us) client_negotiation.cc:240] Negotiation successful
0620 22:56:42.865310 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":45,"thread_start_us":97,"threads_started":1}
I20260620 22:56:42.865640 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:52165
I20260620 22:56:42.865736 8755 heartbeater.cc:347] Connected to a master server at 127.1.166.61:37875
I20260620 22:56:42.865787 8755 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:42.865877 8755 heartbeater.cc:511] Master 127.1.166.61:37875 requested a full tablet report, sending...
I20260620 22:56:42.866231 8381 ts_manager.cc:194] Registered new tserver with Master: 550e61ba7c3542bb9b06f7c1baaf87a2 (127.1.166.1:38073)
I20260620 22:56:42.868650 8753 heartbeater.cc:347] Connected to a master server at 127.1.166.60:38715
I20260620 22:56:42.868745 8753 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:42.868847 8753 heartbeater.cc:511] Master 127.1.166.60:38715 requested a full tablet report, sending...
I20260620 22:56:42.868952 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:41901
--local_ip_for_outbound_sockets=127.1.166.2
--tserver_master_addrs=127.1.166.62:33827
--webserver_port=39809
--webserver_interface=127.1.166.2
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab
--principal=oryx/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:56:42.870582 8574 ts_manager.cc:194] Registered new tserver with Master: 550e61ba7c3542bb9b06f7c1baaf87a2 (127.1.166.1:38073)
W20260620 22:56:42.958787 8761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:42.959017 8761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:42.959054 8761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:42.960322 8761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:42.960371 8761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:42.960409 8761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:56:42.962545 8761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab
--principal=oryx/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=39809
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:42.962821 8761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:42.963054 8761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:42.968585 8769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.968631 8766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:42.968631 8767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:42.969028 8761 server_base.cc:1034] running on GCE node
I20260620 22:56:42.969352 8761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:42.969770 8761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:42.971421 8761 hybrid_clock.cc:648] HybridClock initialized: now 1781996202971395 us; error 37 us; skew 500 ppm
Jun 20 22:56:42 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:42.974761 8761 init.cc:377] Logged in from keytab as oryx/127.1.166.2@KRBTEST.COM (short username oryx)
I20260620 22:56:42.975215 8761 webserver.cc:460] Webserver started at http://127.1.166.2:39809/ using document root <none> and password file <none>
I20260620 22:56:42.975432 8761 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:42.975498 8761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:42.976938 8761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:42.977785 8776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.978125 8761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:42.978245 8761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "42fc7093901745a19824662cacddb9ff"
format_stamp: "Formatted at 2026-06-20 22:56:41 on dist-test-slave-rdl9"
server_key: "0a50a71585533a24036d537ebcbf415e"
server_key_iv: "3f6d3f1d8a439527f1abe2c477d67c6b"
server_key_version: "encryptionkey@0"
I20260620 22:56:42.978698 8761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:42.991441 8761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:42.994963 8761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:42.995155 8761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:42.995448 8761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:42.995838 8761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:42.995875 8761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:42.995911 8761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:42.995935 8761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:43.003408 8761 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:41901
I20260620 22:56:43.003509 8889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:41901 every 8 connection(s)
I20260620 22:56:43.003932 8761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:56:43.004691 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 8761
I20260620 22:56:43.004875 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 8114
I20260620 22:56:43.011310 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:42495
--local_ip_for_outbound_sockets=127.1.166.3
--tserver_master_addrs=127.1.166.62:33827
--webserver_port=35543
--webserver_interface=127.1.166.3
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab
--principal=oryx/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:56:43 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.60@KRBTEST.COM
Jun 20 22:56:43 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.61@KRBTEST.COM
Jun 20 22:56:43 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996202, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:56:43.017624 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.006638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:50977 (local address 127.1.166.60:38715)
0620 22:56:43.006696 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.006698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.007404 (+ 706us) server_negotiation.cc:407] Connection header received
0620 22:56:43.007514 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.007515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.007528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.007558 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.008182 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.008992 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.010672 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.010992 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.011049 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.013993 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.013996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.013996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.014008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.015766 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.016551 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.016552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.016553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.016622 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.016844 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.016844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.016845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.016984 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.017045 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.017408 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.017525 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:56:43.018177 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.005271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:38715 (local address 127.1.166.2:50977)
0620 22:56:43.007338 (+ 2067us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.007348 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.007432 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.007618 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.007618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.007814 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:43.008070 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.008076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.009155 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.009156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.010558 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.010562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.011138 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.011139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.011273 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.011989 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:43.012000 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:43.013824 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:43.016142 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.016154 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.016157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.016475 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.016678 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.016679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.016679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.016788 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.017069 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:43.017076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:43.017256 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.017735 (+ 479us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.017935 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2051,"mutex_wait_us":362,"thread_start_us":76,"threads_started":1}
I20260620 22:56:43.018289 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.005276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:40501 (local address 127.1.166.61:37875)
0620 22:56:43.005328 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.005330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.005631 (+ 301us) server_negotiation.cc:407] Connection header received
0620 22:56:43.006998 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.007000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.007013 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.007041 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.008189 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.008992 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.010709 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.010966 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.011022 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.014814 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.014816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.014817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.014826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.016898 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.017249 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.017250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.017250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.017306 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.017485 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.017486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.017486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.017623 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.017674 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.018124 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.018204 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:56:43.018920 8890 heartbeater.cc:347] Connected to a master server at 127.1.166.60:38715
I20260620 22:56:43.019037 8890 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:43.019122 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.005465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:40833 (local address 127.1.166.62:33827)
0620 22:56:43.005569 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.005572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.006832 (+ 1260us) server_negotiation.cc:407] Connection header received
0620 22:56:43.006966 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.006968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.006983 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.007014 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.008206 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.009000 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.011272 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.011533 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.011778 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.015412 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.015414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.015415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.015424 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.017661 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.018257 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.018258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.018259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.018318 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.018575 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.018576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.018577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.018709 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.018757 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.018959 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.019043 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 22:56:43.019246 8890 heartbeater.cc:511] Master 127.1.166.60:38715 requested a full tablet report, sending...
I20260620 22:56:43.019323 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.004893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.1.166.2:40501)
0620 22:56:43.005444 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.005495 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.006860 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.007226 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.007231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.007731 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:43.008070 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.008076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.009155 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.009156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.010610 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.010613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.011683 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.011684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.011769 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.012731 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:43.012737 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:43.014710 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:43.016982 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.016982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.016983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.017166 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.017368 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.017369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.017369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.017420 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.017953 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:43.017954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:43.018005 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.018927 (+ 922us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.019046 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":160,"thread_start_us":136,"threads_started":1}
I20260620 22:56:43.019596 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.004893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:40833)
0620 22:56:43.006739 (+ 1846us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.006752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.006861 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.007226 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.007231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.007940 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:43.008071 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.008079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.009155 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.009156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.011177 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.011179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.011885 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.011887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.012049 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.012530 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:43.012536 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:43.015261 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:43.017946 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.017948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.017948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.018163 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.018443 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.018444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.018444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.018504 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.018791 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:43.018791 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:43.018839 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.019227 (+ 388us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.019331 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"mutex_wait_us":884,"thread_start_us":127,"threads_started":1}
I20260620 22:56:43.019690 8574 ts_manager.cc:194] Registered new tserver with Master: 42fc7093901745a19824662cacddb9ff (127.1.166.2:41901)
I20260620 22:56:43.019873 8892 heartbeater.cc:347] Connected to a master server at 127.1.166.61:37875
I20260620 22:56:43.019919 8892 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:43.019987 8892 heartbeater.cc:511] Master 127.1.166.61:37875 requested a full tablet report, sending...
I20260620 22:56:43.020093 8891 heartbeater.cc:347] Connected to a master server at 127.1.166.62:33827
I20260620 22:56:43.020130 8891 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:43.020180 8891 heartbeater.cc:511] Master 127.1.166.62:33827 requested a full tablet report, sending...
I20260620 22:56:43.020320 8381 ts_manager.cc:194] Registered new tserver with Master: 42fc7093901745a19824662cacddb9ff (127.1.166.2:41901)
I20260620 22:56:43.020419 7784 ts_manager.cc:194] Re-registered known tserver with Master: 42fc7093901745a19824662cacddb9ff (127.1.166.2:41901)
I20260620 22:56:43.020907 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:40833
W20260620 22:56:43.103812 8897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:43.104079 8897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:43.104106 8897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:43.105408 8897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:43.105473 8897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:43.105505 8897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:56:43.107311 8897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab
--principal=oryx/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:42495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=35543
--tserver_master_addrs=127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:43.107565 8897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:43.107780 8897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:43.113233 8902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:43.113240 8903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:43.113233 8905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:43.113613 8897 server_base.cc:1034] running on GCE node
I20260620 22:56:43.113950 8897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:43.114275 8897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:43.115476 8897 hybrid_clock.cc:648] HybridClock initialized: now 1781996203115456 us; error 47 us; skew 500 ppm
Jun 20 22:56:43 dist-test-slave-rdl9 krb5kdc[7739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996203, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:43.119035 8897 init.cc:377] Logged in from keytab as oryx/127.1.166.3@KRBTEST.COM (short username oryx)
I20260620 22:56:43.119582 8897 webserver.cc:460] Webserver started at http://127.1.166.3:35543/ using document root <none> and password file <none>
I20260620 22:56:43.119812 8897 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:43.119879 8897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:43.121323 8897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:43.122170 8912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:43.122709 8897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:56:43.122853 8897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "2e1a552656224c9facb4281c4d0de28d"
format_stamp: "Formatted at 2026-06-20 22:56:42 on dist-test-slave-rdl9"
server_key: "c67669ba4ef7e92c6b2c10cb1df0bfaf"
server_key_iv: "2e7a9b693c497391393fbb1613ca902d"
server_key_version: "encryptionkey@0"
I20260620 22:56:43.123387 8897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:43.135134 8897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:43.137199 8897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:43.137336 8897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:43.137558 8897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:56:43.137842 8897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:56:43.137874 8897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:43.137904 8897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:56:43.137925 8897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:43.144084 8897 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:42495
I20260620 22:56:43.144134 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:42495 every 8 connection(s)
I20260620 22:56:43.144482 8897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:56:43.150028 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 8897
Jun 20 22:56:43 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996203, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.60@KRBTEST.COM
Jun 20 22:56:43 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996203, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
Jun 20 22:56:43 dist-test-slave-rdl9 krb5kdc[7739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996203, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.61@KRBTEST.COM
I20260620 22:56:43.159291 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.145268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:38715 (local address 127.1.166.3:46111)
0620 22:56:43.145484 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.145531 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.147602 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.147909 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.147913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.148348 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:43.148566 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.148572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.150141 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.150142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.150938 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.150940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.151455 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.151455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.151573 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.152440 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:43.152450 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:43.155009 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:43.157420 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.157431 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.157434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.157788 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.158010 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.158011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.158011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.158159 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.158481 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:43.158489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:43.158680 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.158808 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.159004 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":22,"thread_start_us":125,"threads_started":1}
I20260620 22:56:43.159722 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.145316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:46111 (local address 127.1.166.60:38715)
0620 22:56:43.145386 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.145389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.146536 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 22:56:43.147753 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.147755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.147770 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.147827 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.148811 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.149991 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.151051 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.151341 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.151379 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.155137 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.155139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.155140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.155149 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.157321 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.157868 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.157869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.157870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.157930 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.158225 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.158226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.158226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.158393 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.158453 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.159533 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.159630 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:56:43.160550 9026 heartbeater.cc:347] Connected to a master server at 127.1.166.60:38715
I20260620 22:56:43.160665 9026 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:43.160884 9026 heartbeater.cc:511] Master 127.1.166.60:38715 requested a full tablet report, sending...
I20260620 22:56:43.160964 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.145285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:60039 (local address 127.1.166.61:37875)
0620 22:56:43.145402 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.145405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.146509 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 22:56:43.147841 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.147843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.147856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.147906 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.150194 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.151019 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.151996 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.152279 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.152352 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.155688 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.155689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.155690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.155697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.157536 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.160048 (+ 2512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.160050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.160050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.160116 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.160387 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.160388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.160389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.160543 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.160596 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.160792 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.160871 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 22:56:43.159343 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.145816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:49521 (local address 127.1.166.62:33827)
0620 22:56:43.145852 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.145854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.146505 (+ 651us) server_negotiation.cc:407] Connection header received
0620 22:56:43.147919 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.147921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.147939 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.148020 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.148893 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.149462 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.150957 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.151241 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.151284 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.155672 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.155673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.155674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.155683 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.157839 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.158549 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.158550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.158550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.158610 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.158790 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.158791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.158791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.158929 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.158975 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.159171 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.159255 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:56:43.161274 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.145269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:37875 (local address 127.1.166.3:60039)
0620 22:56:43.145486 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.145532 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.147718 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.147964 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.147965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.148353 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:43.148566 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.148572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.151272 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.151274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.151877 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.151881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.152415 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.152416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.152498 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.152876 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:43.152881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:43.155548 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:43.157615 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.157616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.157616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.159951 (+ 2335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.160239 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.160240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.160241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.160304 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.160633 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:43.160634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:43.160681 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.161051 (+ 370us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.161164 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":8,"thread_start_us":142,"threads_started":1}
I20260620 22:56:43.161405 8574 ts_manager.cc:194] Registered new tserver with Master: 2e1a552656224c9facb4281c4d0de28d (127.1.166.3:42495)
I20260620 22:56:43.161547 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.145667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:49521)
0620 22:56:43.145903 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.145911 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.147614 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.148031 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.148032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.148315 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:43.148777 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.148781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.149594 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.149595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.150842 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.150845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.151392 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.151394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.151573 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.152718 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:43.152724 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:43.155085 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:43.158203 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.158204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.158205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.158461 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.158670 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:43.158670 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:43.158671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:43.158724 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:43.159018 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:43.159018 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:43.159060 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.161327 (+ 2267us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.161434 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":19,"thread_start_us":60,"threads_started":1}
I20260620 22:56:43.161911 9028 heartbeater.cc:347] Connected to a master server at 127.1.166.61:37875
I20260620 22:56:43.161965 9028 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:43.162050 9028 heartbeater.cc:511] Master 127.1.166.61:37875 requested a full tablet report, sending...
I20260620 22:56:43.162396 9027 heartbeater.cc:347] Connected to a master server at 127.1.166.62:33827
I20260620 22:56:43.162442 9027 heartbeater.cc:464] Registering TS with master...
I20260620 22:56:43.162432 8381 ts_manager.cc:194] Registered new tserver with Master: 2e1a552656224c9facb4281c4d0de28d (127.1.166.3:42495)
I20260620 22:56:43.162505 9027 heartbeater.cc:511] Master 127.1.166.62:33827 requested a full tablet report, sending...
I20260620 22:56:43.162701 7784 ts_manager.cc:194] Re-registered known tserver with Master: 2e1a552656224c9facb4281c4d0de28d (127.1.166.3:42495)
I20260620 22:56:43.163061 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.152091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.1.166.62:33827)
0620 22:56:43.152439 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.152442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.152917 (+ 475us) server_negotiation.cc:407] Connection header received
0620 22:56:43.153029 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.153031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.153049 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.153103 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.153873 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.154531 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.156236 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.156580 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.156653 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.159148 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.159150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.159150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.159157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.161160 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.161941 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.161942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.161943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.162022 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.162263 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.162264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.162265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.162423 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.162527 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.162868 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.162961 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":113,"threads_started":1}
I20260620 22:56:43.163192 7784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:49521
I20260620 22:56:43.167007 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.152943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.1.166.61:37875)
0620 22:56:43.155820 (+ 2877us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.155821 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.155828 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:43.155846 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.155847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.155856 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.155917 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.156790 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.157743 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.158792 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.159129 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.159182 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.163822 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.163824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.163824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.163833 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.165441 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.165827 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.165829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.165829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.165893 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.166095 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.166096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.166097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.166235 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.166322 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.166814 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.166905 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2855,"thread_start_us":96,"threads_started":1}
I20260620 22:56:43.165069 7784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:56:43.167923 7784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:56:43.168015 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.155405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.1.166.60:38715)
0620 22:56:43.155984 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.155987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.158102 (+ 2115us) server_negotiation.cc:407] Connection header received
0620 22:56:43.158121 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.158122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.158135 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.160110 (+ 1975us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.160764 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.162480 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.163526 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.163814 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.163876 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.165292 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.165294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.165294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.165303 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.166759 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.167071 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.167072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.167073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.167135 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.167409 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.167410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.167410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.167558 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.167660 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.167841 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.167926 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":56,"threads_started":1}
I20260620 22:56:43.176086 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.171811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:41901 (local address 127.0.0.1:36638)
0620 22:56:43.171946 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.171957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.172011 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.172703 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.172704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.172706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:43.172757 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.172759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.174033 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.174034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.175075 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.175077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.175821 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.175822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.175885 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.175889 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.175943 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.175998 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 22:56:43.176095 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.172091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.1.166.2:41901)
0620 22:56:43.172563 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.172567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.172577 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:43.172615 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.172617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.172632 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.172701 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:43.172842 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.173935 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.175150 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.175742 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.175820 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.175909 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.175962 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.176003 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":104,"threads_started":1}
I20260620 22:56:43.176203 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.171910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:42495 (local address 127.0.0.1:45994)
0620 22:56:43.172367 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.172378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.172422 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.172722 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.172723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.172725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:43.172757 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.172759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.173691 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.173692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.174836 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.174839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.175957 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.175958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.176028 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.176032 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.176077 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.176123 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":243,"threads_started":1}
I20260620 22:56:43.176488 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.172316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.1.166.3:42495)
0620 22:56:43.172490 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.172495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.172506 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:43.172550 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.172552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.172568 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.172655 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:43.172831 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.173598 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.174933 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.175779 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.175847 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.175922 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.176349 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.176397 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260620 22:56:43.176959 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.171910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:38073 (local address 127.0.0.1:54896)
0620 22:56:43.172509 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.172518 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.172556 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.172945 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.172946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.172948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:43.172983 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.172984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.174479 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.174480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.175491 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.175494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.176739 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.176740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.176803 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.176807 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.176846 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.176889 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":5,"thread_start_us":244,"threads_started":1}
I20260620 22:56:43.177076 8824 tablet_service.cc:1467] Processing CreateTablet for tablet d2d7856aa89741ac959ffb6dee51e97a (DEFAULT_TABLE table=test-table [id=60a2691210664ca99a56e98baa27ee0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:43.177075 8960 tablet_service.cc:1467] Processing CreateTablet for tablet d2d7856aa89741ac959ffb6dee51e97a (DEFAULT_TABLE table=test-table [id=60a2691210664ca99a56e98baa27ee0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:43.177237 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.172326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.1.166.1:38073)
0620 22:56:43.172796 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.172799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.172807 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:43.172844 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.172846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.172861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.172937 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:43.173044 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.174330 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.175615 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.176457 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.176520 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.177112 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.177139 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.177180 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":98,"threads_started":1}
I20260620 22:56:43.177464 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d7856aa89741ac959ffb6dee51e97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:43.177464 8960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d7856aa89741ac959ffb6dee51e97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:43.178856 8687 tablet_service.cc:1467] Processing CreateTablet for tablet d2d7856aa89741ac959ffb6dee51e97a (DEFAULT_TABLE table=test-table [id=60a2691210664ca99a56e98baa27ee0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:56:43.179236 8687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d7856aa89741ac959ffb6dee51e97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:43.181640 9052 tablet_bootstrap.cc:492] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: Bootstrap starting.
I20260620 22:56:43.182514 9052 tablet_bootstrap.cc:654] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:43.182833 9052 log.cc:824] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:43.183696 9052 tablet_bootstrap.cc:492] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: No bootstrap required, opened a new log
I20260620 22:56:43.183758 9052 ts_tablet_manager.cc:1397] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:56:43.184981 9055 tablet_bootstrap.cc:492] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Bootstrap starting.
I20260620 22:56:43.185750 9055 tablet_bootstrap.cc:654] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:43.185935 9052 raft_consensus.cc:348] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.186091 9055 log.cc:824] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:43.186124 9052 raft_consensus.cc:374] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:43.186167 9052 raft_consensus.cc:729] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1a552656224c9facb4281c4d0de28d, State: Initialized, Role: FOLLOWER
I20260620 22:56:43.186290 9052 consensus_queue.cc:260] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.186553 9054 tablet_bootstrap.cc:492] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: Bootstrap starting.
I20260620 22:56:43.186585 9052 ts_tablet_manager.cc:1428] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 22:56:43.186719 9027 heartbeater.cc:503] Master 127.1.166.62:33827 was elected leader, sending a full tablet report...
I20260620 22:56:43.187000 9055 tablet_bootstrap.cc:492] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: No bootstrap required, opened a new log
I20260620 22:56:43.187068 9055 ts_tablet_manager.cc:1397] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:56:43.187229 9054 tablet_bootstrap.cc:654] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:43.187837 9054 log.cc:824] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:43.188824 9054 tablet_bootstrap.cc:492] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: No bootstrap required, opened a new log
I20260620 22:56:43.188882 9054 ts_tablet_manager.cc:1397] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:56:43.189457 9055 raft_consensus.cc:348] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.189610 9055 raft_consensus.cc:374] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:43.189631 9055 raft_consensus.cc:729] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42fc7093901745a19824662cacddb9ff, State: Initialized, Role: FOLLOWER
I20260620 22:56:43.189738 9055 consensus_queue.cc:260] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.189973 9055 ts_tablet_manager.cc:1428] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:56:43.190039 8891 heartbeater.cc:503] Master 127.1.166.62:33827 was elected leader, sending a full tablet report...
I20260620 22:56:43.190904 9054 raft_consensus.cc:348] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.191344 9054 raft_consensus.cc:374] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:43.191395 9054 raft_consensus.cc:729] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 550e61ba7c3542bb9b06f7c1baaf87a2, State: Initialized, Role: FOLLOWER
I20260620 22:56:43.191516 9054 consensus_queue.cc:260] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.191797 9054 ts_tablet_manager.cc:1428] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:56:43.191826 8754 heartbeater.cc:503] Master 127.1.166.62:33827 was elected leader, sending a full tablet report...
I20260620 22:56:43.223935 9060 raft_consensus.cc:482] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:56:43.224102 9060 raft_consensus.cc:504] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.224499 9060 leader_election.cc:288] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42fc7093901745a19824662cacddb9ff (127.1.166.2:41901), 2e1a552656224c9facb4281c4d0de28d (127.1.166.3:42495)
I20260620 22:56:43.227406 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.224722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:41901 (local address 127.1.166.1:47793)
0620 22:56:43.224778 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.224792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.224862 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.225136 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.225137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.225139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:43.225200 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.225201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.225781 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.225782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.226626 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.226629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.227175 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.227176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.227240 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.227243 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.227284 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.227325 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260620 22:56:43.227402 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.224799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:47793 (local address 127.1.166.2:41901)
0620 22:56:43.224855 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.224857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.224867 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:43.225027 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.225028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.225039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.225106 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:43.225261 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.225715 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.226707 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.227101 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.227155 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.227169 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.227295 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.227334 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:56:43.227576 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.224866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:35521 (local address 127.1.166.3:42495)
0620 22:56:43.224935 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.224937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.224945 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:43.224971 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.224972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.224983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.225030 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:56:43.225178 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.225642 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.226729 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.227144 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.227207 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.227217 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.227336 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.227376 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:56:43.227910 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.224722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:42495 (local address 127.1.166.1:35521)
0620 22:56:43.224825 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:43.224836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:43.224883 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:43.225040 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:43.225041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:43.225042 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:56:43.225117 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.225119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.225711 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.225712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.226650 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:43.226653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.227215 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.227216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:43.227280 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.227282 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:56:43.227326 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:56:43.227367 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":143}
I20260620 22:56:43.228026 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d7856aa89741ac959ffb6dee51e97a" candidate_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42fc7093901745a19824662cacddb9ff" is_pre_election: true
I20260620 22:56:43.228248 8844 raft_consensus.cc:2454] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 550e61ba7c3542bb9b06f7c1baaf87a2 in term 0.
I20260620 22:56:43.228287 8980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d7856aa89741ac959ffb6dee51e97a" candidate_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1a552656224c9facb4281c4d0de28d" is_pre_election: true
I20260620 22:56:43.228475 8980 raft_consensus.cc:2454] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 550e61ba7c3542bb9b06f7c1baaf87a2 in term 0.
I20260620 22:56:43.228536 8643 leader_election.cc:302] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42fc7093901745a19824662cacddb9ff, 550e61ba7c3542bb9b06f7c1baaf87a2; no voters:
I20260620 22:56:43.228699 9060 raft_consensus.cc:2781] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:56:43.228780 9060 raft_consensus.cc:482] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:56:43.228803 9060 raft_consensus.cc:3037] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:43.229494 9060 raft_consensus.cc:504] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.229630 9060 leader_election.cc:288] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [CANDIDATE]: Term 1 election: Requested vote from peers 42fc7093901745a19824662cacddb9ff (127.1.166.2:41901), 2e1a552656224c9facb4281c4d0de28d (127.1.166.3:42495)
I20260620 22:56:43.229835 8980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d7856aa89741ac959ffb6dee51e97a" candidate_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1a552656224c9facb4281c4d0de28d"
I20260620 22:56:43.229851 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d7856aa89741ac959ffb6dee51e97a" candidate_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42fc7093901745a19824662cacddb9ff"
I20260620 22:56:43.229934 8980 raft_consensus.cc:3037] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:43.229944 8844 raft_consensus.cc:3037] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:43.230861 8844 raft_consensus.cc:2454] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 550e61ba7c3542bb9b06f7c1baaf87a2 in term 1.
I20260620 22:56:43.231061 8643 leader_election.cc:302] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42fc7093901745a19824662cacddb9ff, 550e61ba7c3542bb9b06f7c1baaf87a2; no voters:
I20260620 22:56:43.231158 9060 raft_consensus.cc:2781] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:43.231269 8980 raft_consensus.cc:2454] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 550e61ba7c3542bb9b06f7c1baaf87a2 in term 1.
I20260620 22:56:43.231312 9060 raft_consensus.cc:686] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 1 LEADER]: Becoming Leader. State: Replica: 550e61ba7c3542bb9b06f7c1baaf87a2, State: Running, Role: LEADER
I20260620 22:56:43.231638 9060 consensus_queue.cc:237] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } }
I20260620 22:56:43.232522 7784 catalog_manager.cc:5495] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 550e61ba7c3542bb9b06f7c1baaf87a2 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "550e61ba7c3542bb9b06f7c1baaf87a2" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 } health_report { overall_health: UNKNOWN } } }
I20260620 22:56:43.252159 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.249787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.1.166.1:38073)
0620 22:56:43.249849 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.249851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.249862 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:56:43.249899 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.249900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.249922 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.249983 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:43.250106 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.250640 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.251414 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.251624 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.251685 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.251824 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:43.251929 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:43.251994 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.252062 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.252082 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:56:43.254544 8844 raft_consensus.cc:1261] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 1 FOLLOWER]: Refusing update from remote peer 550e61ba7c3542bb9b06f7c1baaf87a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:43.254539 8980 raft_consensus.cc:1261] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 1 FOLLOWER]: Refusing update from remote peer 550e61ba7c3542bb9b06f7c1baaf87a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 22:56:43.254909 8893 tablet.cc:2369] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:56:43.254923 9061 consensus_queue.cc:1035] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1a552656224c9facb4281c4d0de28d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:43.255072 9060 consensus_queue.cc:1035] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42fc7093901745a19824662cacddb9ff" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:43.258082 9064 mvcc.cc:205] Tried to move back new op lower bound from 7299056448526491648 to 7299056448437698560. Current Snapshot: MvccSnapshot[applied={T|T < 7299056448526491648}]
I20260620 22:56:43.259600 9066 mvcc.cc:205] Tried to move back new op lower bound from 7299056448526491648 to 7299056448437698560. Current Snapshot: MvccSnapshot[applied={T|T < 7299056448526491648}]
I20260620 22:56:43.262367 7784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:43.262471 7784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:56:43.263818 7784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90020d0371ed4eaf9ef69a85170700a2: Sending DeleteTablet for 3 replicas of tablet d2d7856aa89741ac959ffb6dee51e97a
I20260620 22:56:43.264163 8687 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d7856aa89741ac959ffb6dee51e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:43 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:54896
I20260620 22:56:43.264465 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d7856aa89741ac959ffb6dee51e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:43 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:36638
I20260620 22:56:43.264536 8960 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d7856aa89741ac959ffb6dee51e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:56:43 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:45994
I20260620 22:56:43.264765 9077 tablet_replica.cc:330] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: stopping tablet replica
I20260620 22:56:43.264941 9077 raft_consensus.cc:2229] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:43.265121 9077 raft_consensus.cc:2258] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:43.265115 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 7752
I20260620 22:56:43.265851 9077 ts_tablet_manager.cc:1905] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:43.269696 9076 tablet_replica.cc:330] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: stopping tablet replica
I20260620 22:56:43.271004 9077 ts_tablet_manager.cc:1918] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:43.271301 9076 raft_consensus.cc:2229] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:56:43.271497 9076 raft_consensus.cc:2258] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:56:43.272096 9077 log.cc:1196] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/d2d7856aa89741ac959ffb6dee51e97a
I20260620 22:56:43.272428 9075 tablet_replica.cc:330] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: stopping tablet replica
I20260620 22:56:43.273087 9077 ts_tablet_manager.cc:1939] T d2d7856aa89741ac959ffb6dee51e97a P 2e1a552656224c9facb4281c4d0de28d: Deleting consensus metadata
I20260620 22:56:43.273406 9075 raft_consensus.cc:2229] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:56:43.273671 9075 raft_consensus.cc:2258] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:56:43.274319 8915 connection.cc:441] server connection from 127.0.0.1:45994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45994 (request call id 1) could send its response
I20260620 22:56:43.274853 9076 ts_tablet_manager.cc:1905] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:43.276513 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.265087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.1.166.60:38715)
0620 22:56:43.265450 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.265452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.265588 (+ 136us) server_negotiation.cc:407] Connection header received
0620 22:56:43.266486 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.266488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.266504 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.266533 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.268291 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.269095 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.270703 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.271041 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.271092 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.273142 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.273144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.273144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.273156 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.275061 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.275541 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.275543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.275543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.275606 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.275814 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.275815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.275816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.275970 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.276049 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.276255 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.276416 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338}
I20260620 22:56:43.277029 9076 ts_tablet_manager.cc:1918] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:43.277163 9076 log.cc:1196] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/d2d7856aa89741ac959ffb6dee51e97a
I20260620 22:56:43.277506 9076 ts_tablet_manager.cc:1939] T d2d7856aa89741ac959ffb6dee51e97a P 42fc7093901745a19824662cacddb9ff: Deleting consensus metadata
W20260620 22:56:43.277935 8778 connection.cc:441] server connection from 127.0.0.1:36638 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36638 (request call id 1) could send its response
I20260620 22:56:43.278425 9075 ts_tablet_manager.cc:1905] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:56:43.280215 9075 ts_tablet_manager.cc:1918] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:56:43.280439 9075 log.cc:1196] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/d2d7856aa89741ac959ffb6dee51e97a
I20260620 22:56:43.280781 9075 ts_tablet_manager.cc:1939] T d2d7856aa89741ac959ffb6dee51e97a P 550e61ba7c3542bb9b06f7c1baaf87a2: Deleting consensus metadata
W20260620 22:56:43.281246 8643 connection.cc:441] server connection from 127.0.0.1:54896 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54896 (request call id 1) could send its response
I20260620 22:56:43.286895 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.277383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.1.166.60:38715)
0620 22:56:43.277418 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.277420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.277868 (+ 448us) server_negotiation.cc:407] Connection header received
0620 22:56:43.277956 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.277957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.277969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.277997 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.279243 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.280023 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.281234 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.281591 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.281628 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.283578 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.283580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.283580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.283590 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.285425 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.285905 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.285905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.285906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.285966 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.286181 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.286182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.286183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.286318 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.286432 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.286676 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.286773 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:56:43.286896 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:43.277629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.1.166.61:37875)
0620 22:56:43.277674 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:43.277676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:43.277860 (+ 184us) server_negotiation.cc:407] Connection header received
0620 22:56:43.277960 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:43.277961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:43.277977 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:43.278017 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:43.279243 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.280060 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.281235 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:43.281546 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:43.281593 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:43.283617 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:43.283619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:43.283620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:43.283629 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:43.285184 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.285646 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.285647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.285647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.285715 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:43.285954 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:43.285955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:43.285956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:43.286103 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:43.286427 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:43.286630 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 22:56:43.286755 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:44.260617 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:44.259984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:43347)
0620 22:56:44.260436 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:44.260546 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":94,"threads_started":1}
I20260620 22:56:44.260725 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:44.259984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:55211)
0620 22:56:44.260606 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:44.260681 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":95,"threads_started":1}
W20260620 22:56:44.260948 8754 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:33827 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
W20260620 22:56:44.260982 8891 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:33827 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:44.261121 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:44.260392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:44707)
0620 22:56:44.260992 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:44.261078 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":73,"threads_started":1}
W20260620 22:56:44.261313 9027 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:33827 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:44.764840 9089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90020d0371ed4eaf9ef69a85170700a2)
I20260620 22:56:44.765026 9089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:44.765273 9089 leader_election.cc:288] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827)
I20260620 22:56:44.765967 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:44.765736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46636)
0620 22:56:44.765873 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:44.765926 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
W20260620 22:56:44.766129 8366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:44.767936 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:44.767784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46640)
0620 22:56:44.767861 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:44.767899 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 22:56:44.768172 8366 leader_election.cc:334] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827): Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:44.768343 8366 leader_election.cc:302] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 64f56dd320604e1e9d7d50507c83681b; no voters: 90020d0371ed4eaf9ef69a85170700a2
I20260620 22:56:44.768548 9089 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:56:45.261865 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:45.261498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:44799)
0620 22:56:45.261714 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:45.261781 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260620 22:56:45.262039 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:45.261643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:52677)
0620 22:56:45.261914 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:45.261975 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":90,"threads_started":1}
I20260620 22:56:45.262140 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:45.261628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:53659)
0620 22:56:45.262068 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:45.262101 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":52,"threads_started":1}
I20260620 22:56:46.262980 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:46.262729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:60273)
0620 22:56:46.262871 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:46.262916 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260620 22:56:46.263243 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:46.262955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:39485)
0620 22:56:46.263156 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:46.263184 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260620 22:56:46.263243 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:46.262955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:53733)
0620 22:56:46.263135 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:46.263178 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260620 22:56:46.306391 9098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90020d0371ed4eaf9ef69a85170700a2)
I20260620 22:56:46.306553 9098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:46.306716 9098 leader_election.cc:288] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827)
I20260620 22:56:46.307233 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:46.306908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46894)
0620 22:56:46.307113 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:46.307175 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":54,"threads_started":1}
I20260620 22:56:46.307885 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:46.307784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:46908)
0620 22:56:46.307838 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:46.307865 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 22:56:46.307976 8366 leader_election.cc:334] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827): Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:46.308007 8366 leader_election.cc:302] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 64f56dd320604e1e9d7d50507c83681b; no voters: 90020d0371ed4eaf9ef69a85170700a2
I20260620 22:56:46.308079 9098 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:56:47.263891 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:47.263705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:53241)
0620 22:56:47.263806 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:47.263841 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":39,"threads_started":1}
I20260620 22:56:47.263891 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:47.263651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:55053)
0620 22:56:47.263792 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:47.263835 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260620 22:56:47.263974 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:47.263674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:33833)
0620 22:56:47.263901 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:47.263934 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":62,"threads_started":1}
I20260620 22:56:48.264763 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:48.264466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:59997)
0620 22:56:48.264659 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:48.264702 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
I20260620 22:56:48.264901 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:48.264674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:38949)
0620 22:56:48.264809 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:48.264857 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260620 22:56:48.265015 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:48.264466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:41099)
0620 22:56:48.264752 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:48.264795 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":81,"threads_started":1}
I20260620 22:56:49.086964 9107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90020d0371ed4eaf9ef69a85170700a2)
I20260620 22:56:49.087070 9107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:49.087212 9107 leader_election.cc:288] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827)
I20260620 22:56:49.087627 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:49.087433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:49414)
0620 22:56:49.087553 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:49.087590 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260620 22:56:49.087980 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:49.087890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:49422)
0620 22:56:49.087930 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:49.087962 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:56:49.088042 8366 leader_election.cc:334] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827): Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:49.088070 8366 leader_election.cc:302] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 64f56dd320604e1e9d7d50507c83681b; no voters: 90020d0371ed4eaf9ef69a85170700a2
I20260620 22:56:49.088125 9107 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:56:49.265533 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:49.265284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:42169)
0620 22:56:49.265426 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:49.265472 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260620 22:56:49.265542 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:49.265318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:44487)
0620 22:56:49.265446 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:49.265487 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 22:56:49.265971 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:49.265743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:59717)
0620 22:56:49.265876 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:49.265920 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260620 22:56:50.266255 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:50.266042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:47617)
0620 22:56:50.266154 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:50.266196 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260620 22:56:50.266605 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:50.266367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:53375)
0620 22:56:50.266535 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:50.266568 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260620 22:56:50.266651 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:50.266128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:49945)
0620 22:56:50.266313 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:50.266478 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260620 22:56:51.267232 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:51.266980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:42761)
0620 22:56:51.267130 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:51.267174 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260620 22:56:51.267228 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:51.266980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:41625)
0620 22:56:51.267130 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:51.267174 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260620 22:56:51.267514 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:51.267320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:33957)
0620 22:56:51.267433 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:51.267477 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260620 22:56:52.267971 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:52.267739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:57825)
0620 22:56:52.267857 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:52.267902 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260620 22:56:52.268179 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:52.267955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:52471)
0620 22:56:52.268097 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:52.268135 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260620 22:56:52.268371 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:52.268201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:54049)
0620 22:56:52.268300 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:52.268329 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
I20260620 22:56:52.972280 9122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90020d0371ed4eaf9ef69a85170700a2)
I20260620 22:56:52.972399 9122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:52.972530 9122 leader_election.cc:288] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827)
I20260620 22:56:52.972854 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:52.972682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:49692)
0620 22:56:52.972782 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:52.972818 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":45,"threads_started":1}
W20260620 22:56:52.972931 8366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260620 22:56:52.973253 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:52.973170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:49702)
0620 22:56:52.973212 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:52.973236 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:56:52.973322 8366 leader_election.cc:334] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827): Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:52.973361 8366 leader_election.cc:302] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 64f56dd320604e1e9d7d50507c83681b; no voters: 90020d0371ed4eaf9ef69a85170700a2
I20260620 22:56:52.973412 9122 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:56:53.269438 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:53.268890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:50107)
0620 22:56:53.269299 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:53.269355 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":128,"threads_started":1}
I20260620 22:56:53.269654 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:53.269401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:58237)
0620 22:56:53.269551 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:53.269591 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260620 22:56:53.269852 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:53.269607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:45563)
0620 22:56:53.269754 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:53.269801 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":54,"threads_started":1}
I20260620 22:56:54.270464 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:54.270013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:35941)
0620 22:56:54.270309 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:54.270382 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260620 22:56:54.271154 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:54.270510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:38925)
0620 22:56:54.271020 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:54.271093 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":399,"threads_started":1}
I20260620 22:56:54.271310 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:54.270998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:51963)
0620 22:56:54.271120 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:54.271251 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260620 22:56:55.271414 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:55.271113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:39343)
0620 22:56:55.271314 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:55.271357 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
I20260620 22:56:55.271818 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:55.271573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:34007)
0620 22:56:55.271696 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:55.271751 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260620 22:56:55.272006 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:55.271718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:40539)
0620 22:56:55.271905 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:55.271956 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
I20260620 22:56:56.272424 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:56.271980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:46245)
0620 22:56:56.272175 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:56.272219 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260620 22:56:56.272614 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:56.272302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:41601)
0620 22:56:56.272532 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:56.272568 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":154,"threads_started":1}
I20260620 22:56:56.272991 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:56.272628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:47979)
0620 22:56:56.272915 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:56.272952 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":49,"threads_started":1}
I20260620 22:56:57.273263 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:57.272959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:45423)
0620 22:56:57.273140 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:57.273192 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260620 22:56:57.273435 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:57.273207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:54741)
0620 22:56:57.273349 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:57.273385 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":47,"threads_started":1}
I20260620 22:56:57.273640 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:57.273448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:35465)
0620 22:56:57.273563 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:57.273595 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260620 22:56:58.274466 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:58.274193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:37383)
0620 22:56:58.274394 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:58.274423 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260620 22:56:58.274442 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:58.274097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:34569)
0620 22:56:58.274286 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:58.274375 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260620 22:56:58.274457 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:58.274097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:34021)
0620 22:56:58.274313 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:58.274384 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
I20260620 22:56:59.275136 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:59.274916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:53573)
0620 22:56:59.275055 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:59.275084 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":118,"threads_started":1}
I20260620 22:56:59.275136 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:59.274871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:51981)
0620 22:56:59.275043 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:59.275084 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
I20260620 22:56:59.275228 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:59.275025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:47241)
0620 22:56:59.275143 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:59.275178 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":42,"threads_started":1}
I20260620 22:56:59.278420 9146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90020d0371ed4eaf9ef69a85170700a2)
I20260620 22:56:59.278509 9146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:56:59.278617 9146 leader_election.cc:288] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827)
I20260620 22:56:59.278914 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:59.278733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:58312)
0620 22:56:59.278844 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:59.278881 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":45,"threads_started":1}
W20260620 22:56:59.278991 8366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:59.279291 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:59.279213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:58322)
0620 22:56:59.279248 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:59.279274 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 22:56:59.279354 8366 leader_election.cc:334] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827): Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:56:59.279400 8366 leader_election.cc:302] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 64f56dd320604e1e9d7d50507c83681b; no voters: 90020d0371ed4eaf9ef69a85170700a2
I20260620 22:56:59.279475 9146 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:57:00.276005 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:00.275776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:54485)
0620 22:57:00.275902 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:00.275946 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260620 22:57:00.276137 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:00.275779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:60425)
0620 22:57:00.276042 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:00.276087 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260620 22:57:00.276120 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:00.275802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:47527)
0620 22:57:00.275972 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:00.276022 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260620 22:57:01.276978 9154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:01.276683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:39177)
0620 22:57:01.276850 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:01.276901 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260620 22:57:01.277021 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:01.276693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:40369)
0620 22:57:01.276924 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:01.276966 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260620 22:57:01.277396 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:01.276683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:33051)
0620 22:57:01.276864 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:01.276900 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260620 22:57:02.277969 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:02.277717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:58631)
0620 22:57:02.277880 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:02.277913 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260620 22:57:02.277969 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:02.277701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:39533)
0620 22:57:02.277868 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:02.277912 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260620 22:57:02.278182 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:02.277821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:44817)
0620 22:57:02.278089 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:02.278131 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":63,"threads_started":1}
I20260620 22:57:03.278802 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:03.278543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:42851)
0620 22:57:03.278706 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:03.278747 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260620 22:57:03.278803 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:03.278543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:36029)
0620 22:57:03.278701 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:03.278745 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260620 22:57:03.278837 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:03.278629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:50445)
0620 22:57:03.278752 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:03.278783 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260620 22:57:04.038446 9161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90020d0371ed4eaf9ef69a85170700a2)
I20260620 22:57:04.038555 9161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "90020d0371ed4eaf9ef69a85170700a2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33827 } } peers { permanent_uuid: "64f56dd320604e1e9d7d50507c83681b" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 37875 } attrs { promote: false } } peers { permanent_uuid: "f5e6eebf0fea49aa91a389a3331f88a9" member_type: NON_VOTER last_known_addr { host: "127.1.166.60" port: 38715 } attrs { promote: true } }
I20260620 22:57:04.038697 9161 leader_election.cc:288] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827)
I20260620 22:57:04.039099 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:04.038877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:58554)
0620 22:57:04.039024 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:04.039061 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":53,"threads_started":1}
I20260620 22:57:04.039520 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:04.039433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.0.0.1:58566)
0620 22:57:04.039476 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:04.039500 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:57:04.039590 8366 leader_election.cc:334] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90020d0371ed4eaf9ef69a85170700a2 (127.1.166.62:33827): Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
I20260620 22:57:04.039631 8366 leader_election.cc:302] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 64f56dd320604e1e9d7d50507c83681b; no voters: 90020d0371ed4eaf9ef69a85170700a2
I20260620 22:57:04.039692 9161 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 64f56dd320604e1e9d7d50507c83681b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:57:04.279528 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:04.279307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:50673)
0620 22:57:04.279427 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:04.279466 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260620 22:57:04.279556 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:04.279347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:43057)
0620 22:57:04.279466 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:04.279502 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260620 22:57:04.279582 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:04.279347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:42835)
0620 22:57:04.279512 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:04.279539 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260620 22:57:05.280396 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:05.280129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:38111)
0620 22:57:05.280268 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:05.280315 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260620 22:57:05.280414 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:05.280067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:38589)
0620 22:57:05.280307 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:05.280350 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
I20260620 22:57:05.280557 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:05.280067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:50263)
0620 22:57:05.280278 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:05.280328 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260620 22:57:06.281261 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:06.281029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:42915)
0620 22:57:06.281157 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:06.281202 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260620 22:57:06.281316 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:06.281067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:32817)
0620 22:57:06.281208 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:06.281269 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260620 22:57:06.281529 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:06.281067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:36383)
0620 22:57:06.281232 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:06.281264 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260620 22:57:07.281975 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:07.281744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:53259)
0620 22:57:07.281883 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:07.281921 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260620 22:57:07.281975 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:07.281742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:34739)
0620 22:57:07.281886 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:07.281921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260620 22:57:07.282039 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:07.281871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:46967)
0620 22:57:07.281975 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:07.282002 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260620 22:57:08.282702 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:08.282498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:47583)
0620 22:57:08.282613 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:08.282649 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260620 22:57:08.282701 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:08.282476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:56647)
0620 22:57:08.282612 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:08.282647 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260620 22:57:08.282701 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:08.282476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:40219)
0620 22:57:08.282608 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:08.282647 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260620 22:57:09.283388 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:09.283133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:51551)
0620 22:57:09.283286 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:09.283327 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260620 22:57:09.283442 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:09.283133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:48119)
0620 22:57:09.283352 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:09.283392 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260620 22:57:09.283532 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:09.283314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:39627)
0620 22:57:09.283452 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:09.283485 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":49,"threads_started":1}
I20260620 22:57:10.284258 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:10.283986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:51843)
0620 22:57:10.284142 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:10.284198 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260620 22:57:10.284242 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:10.283954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:57631)
0620 22:57:10.284122 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:10.284168 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260620 22:57:10.284242 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:10.283954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:46889)
0620 22:57:10.284122 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:10.284168 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260620 22:57:11.284941 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:11.284693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:34491)
0620 22:57:11.284842 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:11.284882 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260620 22:57:11.284945 9186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:11.284725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:37515)
0620 22:57:11.284842 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:11.284882 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260620 22:57:11.285120 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:11.284821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:46263)
0620 22:57:11.285000 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:11.285046 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260620 22:57:12.285727 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:12.285531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:45769)
0620 22:57:12.285649 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:12.285674 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260620 22:57:12.285727 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:12.285525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:33753)
0620 22:57:12.285640 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:12.285674 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260620 22:57:12.285746 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:12.285467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:55417)
0620 22:57:12.285602 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:12.285704 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260620 22:57:13.286405 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:13.286195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.2:55193)
0620 22:57:13.286293 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:13.286328 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":38,"threads_started":1}
I20260620 22:57:13.286466 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:13.286165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.1:43729)
0620 22:57:13.286305 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:13.286364 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260620 22:57:13.286487 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:13.286288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33827 (local address 127.1.166.3:46941)
0620 22:57:13.286417 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:13.286448 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:33827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715, num_attempts: 242) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.166.62:33827,127.1.166.61:37875,127.1.166.60:38715, num_attempts: 1)
I20260620 22:57:13.287669 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 8528
I20260620 22:57:13.294113 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 8761
I20260620 22:57:13.300028 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 8897
I20260620 22:57:13.305075 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 8248
I20260620 22:57:13.309502 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 8435
2026-06-20T22:57:13Z chronyd exiting
I20260620 22:57:13.326429 1688 test_util.cc:175] -----------------------------------------------
I20260620 22:57:13.326494 1688 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996175763767-1688-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32950 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:57:13 dist-test-slave-rdl9 krb5kdc[9196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:57:13 dist-test-slave-rdl9 krb5kdc[9196](info): set up 2 sockets
Jun 20 22:57:13 dist-test-slave-rdl9 krb5kdc[9196](info): commencing operation
krb5kdc: starting...
W20260620 22:57:14.302744 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:57:14 dist-test-slave-rdl9 krb5kdc[9196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:57:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.62.keytab.
Entry for principal oryx/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.62.keytab.
I20260620 22:57:14.443099 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46797
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.62.keytab
--principal=oryx/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:14.533411 9209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:14.533645 9209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:14.533668 9209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:14.535944 9209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:57:14.536075 9209 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:57:14.536095 9209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:14.536111 9209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:14.536124 9209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:57:14.536137 9209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:57:14.538792 9209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:46797
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.62.keytab
--principal=oryx/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:14.539086 9209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:14.539330 9209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:14.544917 9215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:14.544945 9209 server_base.cc:1034] running on GCE node
W20260620 22:57:14.544905 9214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:14.544917 9217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:14.545624 9209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:14.546005 9209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:14.547197 9209 hybrid_clock.cc:648] HybridClock initialized: now 1781996234547177 us; error 39 us; skew 500 ppm
Jun 20 22:57:14 dist-test-slave-rdl9 krb5kdc[9196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996234, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:14.550676 9209 init.cc:377] Logged in from keytab as oryx/127.1.166.62@KRBTEST.COM (short username oryx)
I20260620 22:57:14.551229 9209 webserver.cc:460] Webserver started at http://127.1.166.62:43183/ using document root <none> and password file <none>
I20260620 22:57:14.551487 9209 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:14.551553 9209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:14.551689 9209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:14.552911 9209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "b45f012cf44c498a90299bb140d52a4a"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "5d448d966b681b72a622093dab81f19d"
server_key_iv: "1e3c4a4da4085c8abaf433a0a2b4c48d"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.553557 9209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "b45f012cf44c498a90299bb140d52a4a"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "5d448d966b681b72a622093dab81f19d"
server_key_iv: "1e3c4a4da4085c8abaf433a0a2b4c48d"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.555510 9209 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 22:57:14.556521 9224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:14.556773 9209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:14.556883 9209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "b45f012cf44c498a90299bb140d52a4a"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "5d448d966b681b72a622093dab81f19d"
server_key_iv: "1e3c4a4da4085c8abaf433a0a2b4c48d"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.556972 9209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:14.570235 9209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:14.573169 9209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:14.573382 9209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:14.579704 9209 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:46797
I20260620 22:57:14.579847 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:46797 every 8 connection(s)
I20260620 22:57:14.580210 9209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:57:14.581063 9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:14.583885 9287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: Bootstrap starting.
I20260620 22:57:14.584606 9287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:14.584920 9287 log.cc:824] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:14.586024 9287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: No bootstrap required, opened a new log
I20260620 22:57:14.588749 9287 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45f012cf44c498a90299bb140d52a4a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46797 } }
I20260620 22:57:14.588971 9287 raft_consensus.cc:374] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:14.589010 9287 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45f012cf44c498a90299bb140d52a4a, State: Initialized, Role: FOLLOWER
I20260620 22:57:14.589195 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 9209
I20260620 22:57:14.589217 9287 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45f012cf44c498a90299bb140d52a4a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46797 } }
I20260620 22:57:14.589299 9287 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:57:14.589303 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:57:14.589332 9287 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:57:14.589401 9287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:14.589498 1688 external_mini_cluster.cc:1428] Setting key 776ea7bc414231588c08231781abdbb7
I20260620 22:57:14.590189 9287 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45f012cf44c498a90299bb140d52a4a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46797 } }
I20260620 22:57:14.590318 9287 leader_election.cc:302] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45f012cf44c498a90299bb140d52a4a; no voters:
I20260620 22:57:14.590571 9287 leader_election.cc:288] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:57:14.590654 9292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:14.590863 9287 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:57:14.591089 9292 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [term 1 LEADER]: Becoming Leader. State: Replica: b45f012cf44c498a90299bb140d52a4a, State: Running, Role: LEADER
I20260620 22:57:14.591326 9292 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45f012cf44c498a90299bb140d52a4a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46797 } }
I20260620 22:57:14.591884 9294 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b45f012cf44c498a90299bb140d52a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45f012cf44c498a90299bb140d52a4a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46797 } } }
I20260620 22:57:14.591995 9294 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:14.592149 9294 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45f012cf44c498a90299bb140d52a4a. Latest consensus state: current_term: 1 leader_uuid: "b45f012cf44c498a90299bb140d52a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45f012cf44c498a90299bb140d52a4a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46797 } } }
I20260620 22:57:14.592679 9294 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a [sys.catalog]: This master's current role is: LEADER
W20260620 22:57:14.592980 9307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:57:14.593053 9307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:57:14.593096 9303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:57:14.593384 9303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:57:14.595484 9303 catalog_manager.cc:1347] Generated new cluster ID: 18487223fc8f41f19b44ae4073c540c9
I20260620 22:57:14.595548 9303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:57:14 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:57:14.603370 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.590193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.1.166.62:46797)
0620 22:57:14.590468 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:14.590480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:14.590525 (+ 45us) server_negotiation.cc:407] Connection header received
0620 22:57:14.590701 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:14.590710 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:14.590774 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:14.591638 (+ 864us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:14.591921 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.593537 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.594326 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.594936 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.595061 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.598235 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:14.598242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:14.598242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:14.598261 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:14.601341 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.601830 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.601834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.601835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.601969 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.602174 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.602175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.602175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.602569 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:14.602770 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:14.602847 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 22:57:14.603123 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260620 22:57:14.603390 9303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:57:14.604264 9303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:57:14.606894 9303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: Generated new TSK 0
I20260620 22:57:14.607220 9303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab.
Entry for principal oryx/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab.
I20260620 22:57:14.641387 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46797
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab
--principal=oryx/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:14.734006 9313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:14.734331 9313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:14.734635 9313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:14.736974 9313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:14.737231 9313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:14.737498 9313 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:57:14.737593 9313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:57:14.740068 9313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab
--principal=oryx/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:14.740412 9313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:14.740671 9313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:57:14.746277 9313 server_base.cc:1034] running on GCE node
W20260620 22:57:14.746325 9321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:14.746289 9318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:14.746325 9319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:14.746896 9313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:14.747210 9313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:14.748361 9313 hybrid_clock.cc:648] HybridClock initialized: now 1781996234748345 us; error 30 us; skew 500 ppm
Jun 20 22:57:14 dist-test-slave-rdl9 krb5kdc[9196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996234, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:14.753818 9313 init.cc:377] Logged in from keytab as oryx/127.1.166.1@KRBTEST.COM (short username oryx)
I20260620 22:57:14.754263 9313 webserver.cc:460] Webserver started at http://127.1.166.1:43819/ using document root <none> and password file <none>
I20260620 22:57:14.754506 9313 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:14.754556 9313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:14.754681 9313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:14.755786 9313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "90a5137ae25b454cb8fea7072f114575"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "3b3334ac7e8128c5d3cc1c58f782530c"
server_key_iv: "607658c3427bf211ab5a3d03ea7768cc"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.756142 9313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "90a5137ae25b454cb8fea7072f114575"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "3b3334ac7e8128c5d3cc1c58f782530c"
server_key_iv: "607658c3427bf211ab5a3d03ea7768cc"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.757609 9313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:57:14.758437 9328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:14.758594 9313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:14.758677 9313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "90a5137ae25b454cb8fea7072f114575"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "3b3334ac7e8128c5d3cc1c58f782530c"
server_key_iv: "607658c3427bf211ab5a3d03ea7768cc"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.758764 9313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:14.780551 9313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:14.783540 9313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:14.783741 9313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:14.784493 9313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:14.784554 9313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:14.784588 9313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:14.784605 9313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:57:14 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996234, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:57:14.794191 9313 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:43043
I20260620 22:57:14.794358 9443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:43043 every 8 connection(s)
I20260620 22:57:14.794719 9313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:57:14.797935 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 9313
I20260620 22:57:14.798091 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:57:14.798385 1688 external_mini_cluster.cc:1428] Setting key 11191e8654ab02eff9e63672dda87926
I20260620 22:57:14.798745 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.785143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46797 (local address 127.1.166.1:38607)
0620 22:57:14.785364 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:14.785423 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:14.786243 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:14.786748 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:14.786755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:14.787280 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:14.787535 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:14.787546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.788651 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.788653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:14.790660 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:14.790667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.791455 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.791456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:14.791657 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.792658 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:14.792672 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:14.794760 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:14.796631 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:14.796640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:14.796643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:14.796985 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:14.797371 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:14.797372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:14.797373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:14.797518 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:14.797818 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:14.797825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:14.798051 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 22:57:14.798188 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 22:57:14.798467 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20260620 22:57:14.798803 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.785143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:38607 (local address 127.1.166.62:46797)
0620 22:57:14.785205 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:14.785207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:14.785545 (+ 338us) server_negotiation.cc:407] Connection header received
0620 22:57:14.786468 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:14.786469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:14.786490 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:14.786588 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:14.787678 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.788473 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.790877 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.791313 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.791422 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.794933 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:14.794935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:14.794936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:14.794947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:14.796490 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.797158 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.797160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.797160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.797251 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.797600 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.797600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.797600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.797733 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:14.797807 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:14.798544 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 22:57:14.798676 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:57:14.799590 9444 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46797
I20260620 22:57:14.799716 9444 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:14.799961 9444 heartbeater.cc:511] Master 127.1.166.62:46797 requested a full tablet report, sending...
I20260620 22:57:14.801726 9239 ts_manager.cc:194] Registered new tserver with Master: 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043)
I20260620 22:57:14.802841 9239 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:38607
WARNING: no policy specified for oryx/127.1.166.2@KRBTEST.COM; defaulting to no policy
I20260620 22:57:14.808290 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.801317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:50799 (local address 127.1.166.62:46797)
0620 22:57:14.801388 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:14.801390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:14.801398 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:14.801424 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:14.801425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:14.801437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:14.801497 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:14.802073 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.802915 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.803915 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.804128 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.804190 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.805323 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:14.805325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:14.805325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:14.805334 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:14.806963 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.807402 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.807404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.807404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.807473 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.807668 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.807669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.807669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.807803 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:14.807872 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:14.808095 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 22:57:14.808192 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:57:14.808307 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.800843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46797 (local address 127.1.166.1:50799)
0620 22:57:14.801100 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:14.801115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:14.801174 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:14.801512 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:14.801513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:14.801774 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:14.801971 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:14.801975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.803064 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.803066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:14.803824 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:14.803828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.804199 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.804200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:14.804266 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.804836 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:14.804841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:14.805198 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:14.807101 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:14.807103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:14.807104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:14.807322 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:14.807538 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:14.807539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:14.807540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:14.807610 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:14.807869 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:14.807870 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:14.807934 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 22:57:14.808012 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:57:14.808162 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab.
Entry for principal oryx/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab.
I20260620 22:57:14.825407 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46797
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab
--principal=oryx/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:14.931418 9453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:14.931658 9453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:14.931694 9453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:14.933133 9453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:14.933219 9453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:14.933255 9453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:57:14.933282 9453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:57:14.935245 9453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab
--principal=oryx/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:14.935570 9453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:14.935825 9453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:14.940853 9461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:14.940876 9458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:14.940862 9459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:14.941347 9453 server_base.cc:1034] running on GCE node
I20260620 22:57:14.941727 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:14.942118 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:14.943341 9453 hybrid_clock.cc:648] HybridClock initialized: now 1781996234943304 us; error 56 us; skew 500 ppm
Jun 20 22:57:14 dist-test-slave-rdl9 krb5kdc[9196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996234, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:14.947062 9453 init.cc:377] Logged in from keytab as oryx/127.1.166.2@KRBTEST.COM (short username oryx)
I20260620 22:57:14.947678 9453 webserver.cc:460] Webserver started at http://127.1.166.2:41757/ using document root <none> and password file <none>
I20260620 22:57:14.947949 9453 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:14.948004 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:14.948148 9453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:14.949447 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "183c090687e24c8b9fdb66563fd8dd7b"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "2b2122a72a30fe22e29a4ce5a32857d3"
server_key_iv: "a9f17beac6f9051f61cf911bbde38054"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.949808 9453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "183c090687e24c8b9fdb66563fd8dd7b"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "2b2122a72a30fe22e29a4ce5a32857d3"
server_key_iv: "a9f17beac6f9051f61cf911bbde38054"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.951576 9453 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 22:57:14.952580 9468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:14.952795 9453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:14.952986 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "183c090687e24c8b9fdb66563fd8dd7b"
format_stamp: "Formatted at 2026-06-20 22:57:14 on dist-test-slave-rdl9"
server_key: "2b2122a72a30fe22e29a4ce5a32857d3"
server_key_iv: "a9f17beac6f9051f61cf911bbde38054"
server_key_version: "encryptionkey@0"
I20260620 22:57:14.953094 9453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:14.970222 9453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:14.972675 9453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:14.972834 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:14.973320 9453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:14.973361 9453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:14.973392 9453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:14.973408 9453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:14.980384 9453 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:42853
I20260620 22:57:14.980504 9583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:42853 every 8 connection(s)
I20260620 22:57:14.980784 9453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:57:14 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996234, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:57:14.981482 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 9453
I20260620 22:57:14.981590 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:57:14.981807 1688 external_mini_cluster.cc:1428] Setting key 010b088d001ad408c8b066cf89027df9
I20260620 22:57:14.984513 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.973801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:33379 (local address 127.1.166.62:46797)
0620 22:57:14.973866 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:14.973868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:14.974049 (+ 181us) server_negotiation.cc:407] Connection header received
0620 22:57:14.974843 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:14.974844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:14.974864 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:14.974936 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:14.975936 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.976719 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.978384 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.978600 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.978670 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.981513 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:14.981515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:14.981515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:14.981523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:14.982990 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.983481 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.983482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.983482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.983552 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.983843 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.983843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.983844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.983991 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:14.984059 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:14.984327 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 22:57:14.984416 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:57:14.984611 9477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.973761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46797 (local address 127.1.166.2:33379)
0620 22:57:14.973915 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:14.973965 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:14.974695 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:14.975081 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:14.975087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:14.975476 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:14.975801 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:14.975809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.976834 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.976835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:14.978244 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:14.978249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.978692 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.978693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:14.978869 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.979548 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:14.979560 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:14.981374 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:14.983079 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:14.983089 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:14.983091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:14.983389 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:14.983629 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:14.983630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:14.983630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:14.983777 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:14.984070 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:14.984075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:14.984205 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 22:57:14.984293 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:57:14.984427 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260620 22:57:14.985082 9584 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46797
I20260620 22:57:14.985178 9584 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:14.985369 9584 heartbeater.cc:511] Master 127.1.166.62:46797 requested a full tablet report, sending...
I20260620 22:57:14.985772 9239 ts_manager.cc:194] Registered new tserver with Master: 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853)
I20260620 22:57:14.986204 9239 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:33379
WARNING: no policy specified for oryx/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.3@KRBTEST.COM" created.
I20260620 22:57:14.996661 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.986525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46797 (local address 127.1.166.2:41609)
0620 22:57:14.990401 (+ 3876us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:14.990417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:14.990508 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:14.990656 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:14.990657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:14.990869 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:14.990961 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:14.990963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.991702 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.991703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:14.992432 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:14.992435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.992862 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.992863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:14.992935 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.993454 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:14.993459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:14.993899 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:14.995469 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:14.995470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:14.995470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:14.995639 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:14.995946 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:14.995947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:14.995947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:14.996011 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:14.996320 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:14.996321 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:14.996375 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 22:57:14.996435 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:57:14.996555 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3828,"thread_start_us":77,"threads_started":1}
I20260620 22:57:14.996658 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:14.986574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:41609 (local address 127.1.166.62:46797)
0620 22:57:14.986617 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:14.986619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:14.990499 (+ 3880us) server_negotiation.cc:407] Connection header received
0620 22:57:14.990580 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:14.990581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:14.990594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:14.990645 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:14.991036 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.991613 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.992501 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:14.992743 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:14.992808 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:14.994014 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:14.994015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:14.994015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:14.994022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:14.995378 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.995703 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.995753 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.995753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.995843 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:14.996071 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:14.996072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:14.996072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:14.996250 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:14.996316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:14.996463 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 22:57:14.996552 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab.
Entry for principal oryx/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab.
I20260620 22:57:15.009140 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46797
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab
--principal=oryx/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:15.098083 9593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:15.098332 9593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:15.098387 9593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:15.099754 9593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:15.099815 9593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:15.099838 9593 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:57:15.099855 9593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:57:15.101621 9593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab
--principal=oryx/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.166.62:46797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:15.101881 9593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:15.102113 9593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:15.107544 9599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:15.107684 9598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:15.107779 9593 server_base.cc:1034] running on GCE node
W20260620 22:57:15.107864 9601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:15.108088 9593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:15.108358 9593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:15.109576 9593 hybrid_clock.cc:648] HybridClock initialized: now 1781996235109525 us; error 61 us; skew 500 ppm
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996235, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:15.112982 9593 init.cc:377] Logged in from keytab as oryx/127.1.166.3@KRBTEST.COM (short username oryx)
I20260620 22:57:15.113520 9593 webserver.cc:460] Webserver started at http://127.1.166.3:42769/ using document root <none> and password file <none>
I20260620 22:57:15.113776 9593 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:15.113829 9593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:15.113958 9593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:15.115170 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "ec2348b8cad04e40839875b035f6878b"
format_stamp: "Formatted at 2026-06-20 22:57:15 on dist-test-slave-rdl9"
server_key: "8d8c54ecc3973882df045c114a151e20"
server_key_iv: "ec30f31b56ce2194fc34975739e54d75"
server_key_version: "encryptionkey@0"
I20260620 22:57:15.115523 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "ec2348b8cad04e40839875b035f6878b"
format_stamp: "Formatted at 2026-06-20 22:57:15 on dist-test-slave-rdl9"
server_key: "8d8c54ecc3973882df045c114a151e20"
server_key_iv: "ec30f31b56ce2194fc34975739e54d75"
server_key_version: "encryptionkey@0"
I20260620 22:57:15.116991 9593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:57:15.117903 9608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.118134 9593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:15.118239 9593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "ec2348b8cad04e40839875b035f6878b"
format_stamp: "Formatted at 2026-06-20 22:57:15 on dist-test-slave-rdl9"
server_key: "8d8c54ecc3973882df045c114a151e20"
server_key_iv: "ec30f31b56ce2194fc34975739e54d75"
server_key_version: "encryptionkey@0"
I20260620 22:57:15.118371 9593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:15.128329 9593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:15.131175 9593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:15.131363 9593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:15.131934 9593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:15.131978 9593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.132011 9593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:15.132030 9593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996235, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:57:15.139508 9593 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:36511
I20260620 22:57:15.139662 9723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:36511 every 8 connection(s)
I20260620 22:57:15.139968 9593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:57:15.144138 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.132707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:40771 (local address 127.1.166.62:46797)
0620 22:57:15.132782 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.132784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.133064 (+ 280us) server_negotiation.cc:407] Connection header received
0620 22:57:15.133650 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.133651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.133664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.133751 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:15.134728 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.135512 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.137178 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.137423 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.137490 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.140261 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:15.140264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:15.140265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:15.140277 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:15.142043 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.142708 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.142709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.142710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.142783 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.143174 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.143175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.143176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.143461 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:15.143561 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.143955 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.144056 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:57:15.144311 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.132667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46797 (local address 127.1.166.3:40771)
0620 22:57:15.132915 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.132957 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.133547 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.133872 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.133878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.134286 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:15.134628 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.134635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.135617 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.135618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.137042 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.137046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.137514 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.137515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.137660 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.138259 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:15.138268 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:15.140078 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:15.142184 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:15.142193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:15.142196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:15.142596 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:15.142897 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:15.142899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:15.142899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:15.143066 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:15.143573 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:15.143580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:15.143751 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.143870 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.144061 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260620 22:57:15.144702 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 9593
I20260620 22:57:15.144797 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:57:15.145004 1688 external_mini_cluster.cc:1428] Setting key a7a67ec6e9bd12a8f52e763b603f340a
I20260620 22:57:15.144951 9724 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46797
I20260620 22:57:15.145071 9724 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:15.145481 9724 heartbeater.cc:511] Master 127.1.166.62:46797 requested a full tablet report, sending...
I20260620 22:57:15.145992 9239 ts_manager.cc:194] Registered new tserver with Master: ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511)
I20260620 22:57:15.146582 9239 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:40771
I20260620 22:57:15.152768 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.146534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46797 (local address 127.1.166.3:34223)
0620 22:57:15.146661 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.146673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.146729 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.146908 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.146909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.147147 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:15.147225 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.147228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.148031 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.148033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.148632 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.148634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.149102 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.149104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.149189 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.149665 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:15.149670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:15.150009 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:15.151639 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:15.151645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:15.151646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:15.151860 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:15.152079 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:15.152080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:15.152080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:15.152133 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:15.152410 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:15.152412 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:15.152470 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.152539 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.152660 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260620 22:57:15.152774 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.146608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:34223 (local address 127.1.166.62:46797)
0620 22:57:15.146700 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.146702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.146713 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:15.146815 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.146816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.146830 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.146884 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:15.147350 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.147890 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.148763 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.148986 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.149062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.150114 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:15.150115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:15.150116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:15.150128 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:15.151525 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.151957 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.151958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.151959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.152023 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.152189 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.152189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.152190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.152314 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:15.152366 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.152619 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.152700 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 22:57:15.157516 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:57:15.180018 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:15.173559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.1.166.62:46797)
0620 22:57:15.173614 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.173615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.173623 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:15.173651 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.173652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.173660 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.173728 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:15.174636 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.175137 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.176024 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.176227 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.176296 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.179926 (+ 3630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34964: BlockingRecv error: recv got EOF from 127.0.0.1:34964 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:57:15.189108 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.181099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.1.166.62:46797)
0620 22:57:15.181181 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.181183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.181269 (+ 86us) server_negotiation.cc:407] Connection header received
0620 22:57:15.181307 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.181308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.181320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.181397 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:15.181993 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.182585 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.183337 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.183539 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.183615 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.186001 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:15.186002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:15.186002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:15.186012 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:15.187700 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.188197 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.188199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.188199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.188263 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.188521 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.188522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.188522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.188656 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:15.188753 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.188961 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.189035 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 22:57:15.191049 9239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:57:15.192977 9239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:57:15.199965 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.195588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:36511 (local address 127.0.0.1:40444)
0620 22:57:15.195704 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.195745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.195831 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.196349 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.196350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.196352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.196420 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.196421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.197686 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.197687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.198607 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.198609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.199598 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.199599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.199688 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.199725 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.199800 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.199864 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
I20260620 22:57:15.200078 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.195853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.1.166.1:43043)
0620 22:57:15.196010 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.196015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.196057 (+ 42us) server_negotiation.cc:407] Connection header received
0620 22:57:15.196154 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.196156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.196169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.196241 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.196480 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.197744 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.199008 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.199716 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.199777 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.199851 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.199934 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.199976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260620 22:57:15.200183 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.195813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42853 (local address 127.0.0.1:41554)
0620 22:57:15.196216 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.196225 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.196270 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.196502 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.196503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.196505 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.196539 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.196540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.197576 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.197577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.199103 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.199105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.199976 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.199977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.200032 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.200036 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.200080 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.200121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260620 22:57:15.200428 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.196034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.1.166.2:42853)
0620 22:57:15.196362 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.196366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.196375 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:57:15.196411 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.196413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.196427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.196502 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.196591 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.197497 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.199203 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.199817 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.199871 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.199944 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.200283 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.200327 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":80,"threads_started":1}
I20260620 22:57:15.200423 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.195830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.1.166.3:36511)
0620 22:57:15.196203 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.196208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.196219 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:15.196261 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.196263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.196279 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.196353 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.196490 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.197575 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.198682 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.199504 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.199586 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.199670 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.200227 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.200322 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":75,"threads_started":1}
I20260620 22:57:15.200742 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.195597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:43043 (local address 127.0.0.1:38712)
0620 22:57:15.195989 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.195999 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.196071 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.196268 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.196271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.196273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.196354 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.196356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.197821 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.197822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.198918 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.198921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.199793 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.199794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.199858 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.199862 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.199919 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.199966 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":8,"thread_start_us":99,"threads_started":1}
I20260620 22:57:15.201189 9518 tablet_service.cc:1467] Processing CreateTablet for tablet 325b08e8e83644ea9f0653fc92e3ab4a (DEFAULT_TABLE table=test-table [id=d2ce78a01889448cbc387481fbb81610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.201161 9658 tablet_service.cc:1467] Processing CreateTablet for tablet 325b08e8e83644ea9f0653fc92e3ab4a (DEFAULT_TABLE table=test-table [id=d2ce78a01889448cbc387481fbb81610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.201371 9378 tablet_service.cc:1467] Processing CreateTablet for tablet 325b08e8e83644ea9f0653fc92e3ab4a (DEFAULT_TABLE table=test-table [id=d2ce78a01889448cbc387481fbb81610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.201560 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325b08e8e83644ea9f0653fc92e3ab4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.201560 9658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325b08e8e83644ea9f0653fc92e3ab4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.201738 9378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325b08e8e83644ea9f0653fc92e3ab4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.205801 9752 tablet_bootstrap.cc:492] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Bootstrap starting.
I20260620 22:57:15.206118 9754 tablet_bootstrap.cc:492] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Bootstrap starting.
I20260620 22:57:15.206965 9752 tablet_bootstrap.cc:654] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.206977 9754 tablet_bootstrap.cc:654] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.206983 9753 tablet_bootstrap.cc:492] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: Bootstrap starting.
I20260620 22:57:15.207333 9752 log.cc:824] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:15.207372 9754 log.cc:824] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:15.207639 9753 tablet_bootstrap.cc:654] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.207947 9753 log.cc:824] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:15.208243 9754 tablet_bootstrap.cc:492] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: No bootstrap required, opened a new log
I20260620 22:57:15.208271 9752 tablet_bootstrap.cc:492] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: No bootstrap required, opened a new log
I20260620 22:57:15.208316 9754 ts_tablet_manager.cc:1397] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:57:15.208359 9752 ts_tablet_manager.cc:1397] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:57:15.208707 9753 tablet_bootstrap.cc:492] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: No bootstrap required, opened a new log
I20260620 22:57:15.208774 9753 ts_tablet_manager.cc:1397] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:57:15.209910 9754 raft_consensus.cc:348] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.210070 9754 raft_consensus.cc:374] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.210115 9754 raft_consensus.cc:729] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183c090687e24c8b9fdb66563fd8dd7b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.210270 9754 consensus_queue.cc:260] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.210604 9754 ts_tablet_manager.cc:1428] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:57:15.210572 9752 raft_consensus.cc:348] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.210747 9752 raft_consensus.cc:374] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.210811 9752 raft_consensus.cc:729] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a5137ae25b454cb8fea7072f114575, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.210810 9753 raft_consensus.cc:348] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.210955 9753 raft_consensus.cc:374] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.210932 9752 consensus_queue.cc:260] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.210994 9753 raft_consensus.cc:729] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2348b8cad04e40839875b035f6878b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.211102 9753 consensus_queue.cc:260] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.211215 9752 ts_tablet_manager.cc:1428] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260620 22:57:15.211248 9444 heartbeater.cc:503] Master 127.1.166.62:46797 was elected leader, sending a full tablet report...
I20260620 22:57:15.211354 9753 ts_tablet_manager.cc:1428] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:57:15.211462 9724 heartbeater.cc:503] Master 127.1.166.62:46797 was elected leader, sending a full tablet report...
I20260620 22:57:15.211475 9584 heartbeater.cc:503] Master 127.1.166.62:46797 was elected leader, sending a full tablet report...
W20260620 22:57:15.231391 9585 tablet.cc:2369] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:57:15.295732 9445 tablet.cc:2369] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:57:15.312472 9759 raft_consensus.cc:482] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:57:15.312609 9759 raft_consensus.cc:504] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.312984 9759 leader_election.cc:288] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511), 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853)
I20260620 22:57:15.315999 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.313317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:39591 (local address 127.1.166.2:42853)
0620 22:57:15.313362 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.313364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.313417 (+ 53us) server_negotiation.cc:407] Connection header received
0620 22:57:15.313516 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.313517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.313529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.313572 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.313692 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.314161 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.315292 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.315648 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.315718 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.315730 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.315874 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.315913 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:57:15.316010 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.313345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:37721 (local address 127.1.166.3:36511)
0620 22:57:15.313408 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.313409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.313421 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:57:15.313450 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.313451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.313460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.313518 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.313667 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.314171 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.315293 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.315666 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.315728 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.315739 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.315896 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.315935 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:57:15.316036 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.313195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42853 (local address 127.1.166.1:39591)
0620 22:57:15.313328 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.313341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.313441 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.313585 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.313585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.313588 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.313640 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.313642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.314254 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.314255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.315187 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.315191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.315729 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.315730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.315809 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.315813 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.315863 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.315912 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":53,"thread_start_us":72,"threads_started":1}
I20260620 22:57:15.316042 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.313196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:36511 (local address 127.1.166.1:37721)
0620 22:57:15.313330 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.313341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.313401 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.313534 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.313536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.313538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.313607 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.313609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.314255 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.314256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.315190 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.315193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.315743 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.315744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.315831 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.315835 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.315884 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.315929 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":54,"thread_start_us":65,"threads_started":1}
I20260620 22:57:15.316648 9678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325b08e8e83644ea9f0653fc92e3ab4a" candidate_uuid: "90a5137ae25b454cb8fea7072f114575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2348b8cad04e40839875b035f6878b" is_pre_election: true
I20260620 22:57:15.316645 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325b08e8e83644ea9f0653fc92e3ab4a" candidate_uuid: "90a5137ae25b454cb8fea7072f114575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183c090687e24c8b9fdb66563fd8dd7b" is_pre_election: true
I20260620 22:57:15.316793 9678 raft_consensus.cc:2454] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a5137ae25b454cb8fea7072f114575 in term 0.
I20260620 22:57:15.316807 9538 raft_consensus.cc:2454] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a5137ae25b454cb8fea7072f114575 in term 0.
I20260620 22:57:15.317018 9330 leader_election.cc:302] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a5137ae25b454cb8fea7072f114575, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.317140 9759 raft_consensus.cc:2781] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:57:15.317202 9759 raft_consensus.cc:482] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:57:15.317232 9759 raft_consensus.cc:3037] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.317945 9759 raft_consensus.cc:504] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.318076 9759 leader_election.cc:288] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [CANDIDATE]: Term 1 election: Requested vote from peers ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511), 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853)
I20260620 22:57:15.318233 9678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325b08e8e83644ea9f0653fc92e3ab4a" candidate_uuid: "90a5137ae25b454cb8fea7072f114575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2348b8cad04e40839875b035f6878b"
I20260620 22:57:15.318257 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325b08e8e83644ea9f0653fc92e3ab4a" candidate_uuid: "90a5137ae25b454cb8fea7072f114575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183c090687e24c8b9fdb66563fd8dd7b"
I20260620 22:57:15.318315 9678 raft_consensus.cc:3037] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.318323 9538 raft_consensus.cc:3037] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.319016 9538 raft_consensus.cc:2454] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a5137ae25b454cb8fea7072f114575 in term 1.
I20260620 22:57:15.319056 9678 raft_consensus.cc:2454] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a5137ae25b454cb8fea7072f114575 in term 1.
I20260620 22:57:15.319226 9330 leader_election.cc:302] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a5137ae25b454cb8fea7072f114575, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.319355 9759 raft_consensus.cc:2781] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:15.319473 9759 raft_consensus.cc:686] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 1 LEADER]: Becoming Leader. State: Replica: 90a5137ae25b454cb8fea7072f114575, State: Running, Role: LEADER
I20260620 22:57:15.319567 9759 consensus_queue.cc:237] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } }
I20260620 22:57:15.320485 9241 catalog_manager.cc:5495] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a5137ae25b454cb8fea7072f114575 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "90a5137ae25b454cb8fea7072f114575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } health_report { overall_health: UNKNOWN } } }
I20260620 22:57:15.327690 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.325416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.1.166.1:43043)
0620 22:57:15.325481 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.325483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.325491 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:15.325525 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.325526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.325546 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.325611 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:15.325749 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.326226 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.326982 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.327183 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.327236 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.327385 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.327491 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.327567 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.327591 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.327610 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:57:15.329535 9678 raft_consensus.cc:1261] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Refusing update from remote peer 90a5137ae25b454cb8fea7072f114575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:15.329535 9538 raft_consensus.cc:1261] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Refusing update from remote peer 90a5137ae25b454cb8fea7072f114575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:15.329864 9759 consensus_queue.cc:1035] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:15.330024 9763 consensus_queue.cc:1035] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:15.332610 9768 mvcc.cc:205] Tried to move back new op lower bound from 7299056579907035136 to 7299056579869605888. Current Snapshot: MvccSnapshot[applied={T|T < 7299056579907035136}]
I20260620 22:57:15.332849 9767 mvcc.cc:205] Tried to move back new op lower bound from 7299056579907035136 to 7299056579869605888. Current Snapshot: MvccSnapshot[applied={T|T < 7299056579907035136}]
I20260620 22:57:15.334712 9239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:15.334800 9239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:15.335479 9239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: Sending DeleteTablet for 3 replicas of tablet 325b08e8e83644ea9f0653fc92e3ab4a
I20260620 22:57:15.335799 9658 tablet_service.cc:1514] Processing DeleteTablet for tablet 325b08e8e83644ea9f0653fc92e3ab4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:40444
I20260620 22:57:15.335837 9518 tablet_service.cc:1514] Processing DeleteTablet for tablet 325b08e8e83644ea9f0653fc92e3ab4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:41554
I20260620 22:57:15.335886 9378 tablet_service.cc:1514] Processing DeleteTablet for tablet 325b08e8e83644ea9f0653fc92e3ab4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38712
I20260620 22:57:15.335988 9239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:57:15.336113 9239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:57:15.336148 9776 tablet_replica.cc:330] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: stopping tablet replica
I20260620 22:57:15.336236 9776 raft_consensus.cc:2229] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:15.336585 9776 raft_consensus.cc:2258] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.337368 9776 ts_tablet_manager.cc:1905] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.337436 9658 tablet_service.cc:1467] Processing CreateTablet for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa (DEFAULT_TABLE table=test-table [id=41e4388fa8a24c368f155c0d1a23a32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.337450 9518 tablet_service.cc:1467] Processing CreateTablet for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa (DEFAULT_TABLE table=test-table [id=41e4388fa8a24c368f155c0d1a23a32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.337471 9378 tablet_service.cc:1467] Processing CreateTablet for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa (DEFAULT_TABLE table=test-table [id=41e4388fa8a24c368f155c0d1a23a32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.337538 9658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.337572 9378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.337572 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.338765 9753 tablet_bootstrap.cc:492] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: Bootstrap starting.
I20260620 22:57:15.338932 9752 tablet_bootstrap.cc:492] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: Bootstrap starting.
I20260620 22:57:15.338984 9754 tablet_bootstrap.cc:492] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: Bootstrap starting.
I20260620 22:57:15.339154 9753 tablet_bootstrap.cc:654] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.339322 9752 tablet_bootstrap.cc:654] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.339345 9754 tablet_bootstrap.cc:654] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.339500 9776 ts_tablet_manager.cc:1918] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:57:15.339557 9776 log.cc:1196] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/325b08e8e83644ea9f0653fc92e3ab4a
I20260620 22:57:15.339720 9753 tablet_bootstrap.cc:492] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: No bootstrap required, opened a new log
I20260620 22:57:15.339761 9753 ts_tablet_manager.cc:1397] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:15.339900 9753 raft_consensus.cc:348] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.339970 9753 raft_consensus.cc:374] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.339989 9753 raft_consensus.cc:729] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2348b8cad04e40839875b035f6878b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.340031 9753 consensus_queue.cc:260] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.340095 9777 tablet_replica.cc:330] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: stopping tablet replica
I20260620 22:57:15.340119 9753 ts_tablet_manager.cc:1428] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:57:15.340193 9777 raft_consensus.cc:2229] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:15.340304 9777 raft_consensus.cc:2258] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.340504 9776 ts_tablet_manager.cc:1939] T 325b08e8e83644ea9f0653fc92e3ab4a P ec2348b8cad04e40839875b035f6878b: Deleting consensus metadata
I20260620 22:57:15.340783 9777 ts_tablet_manager.cc:1905] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.340909 9226 catalog_manager.cc:4841] TS ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511): tablet 325b08e8e83644ea9f0653fc92e3ab4a (table test-table [id=d2ce78a01889448cbc387481fbb81610]) successfully deleted
I20260620 22:57:15.342007 9777 ts_tablet_manager.cc:1918] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:57:15.342064 9778 tablet_replica.cc:330] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: stopping tablet replica
I20260620 22:57:15.342099 9777 log.cc:1196] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/325b08e8e83644ea9f0653fc92e3ab4a
I20260620 22:57:15.342180 9752 tablet_bootstrap.cc:492] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: No bootstrap required, opened a new log
I20260620 22:57:15.342188 9778 raft_consensus.cc:2229] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:57:15.342236 9752 ts_tablet_manager.cc:1397] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260620 22:57:15.342375 9778 raft_consensus.cc:2258] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.342422 9777 ts_tablet_manager.cc:1939] T 325b08e8e83644ea9f0653fc92e3ab4a P 183c090687e24c8b9fdb66563fd8dd7b: Deleting consensus metadata
I20260620 22:57:15.342399 9752 raft_consensus.cc:348] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.342463 9752 raft_consensus.cc:374] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.342481 9752 raft_consensus.cc:729] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a5137ae25b454cb8fea7072f114575, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.342521 9752 consensus_queue.cc:260] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.342603 9752 ts_tablet_manager.cc:1428] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.342797 9227 catalog_manager.cc:4841] TS 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853): tablet 325b08e8e83644ea9f0653fc92e3ab4a (table test-table [id=d2ce78a01889448cbc387481fbb81610]) successfully deleted
I20260620 22:57:15.342887 9778 ts_tablet_manager.cc:1905] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.344180 9778 ts_tablet_manager.cc:1918] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:57:15.344259 9778 log.cc:1196] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/325b08e8e83644ea9f0653fc92e3ab4a
I20260620 22:57:15.344512 9778 ts_tablet_manager.cc:1939] T 325b08e8e83644ea9f0653fc92e3ab4a P 90a5137ae25b454cb8fea7072f114575: Deleting consensus metadata
I20260620 22:57:15.344898 9227 catalog_manager.cc:4841] TS 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043): tablet 325b08e8e83644ea9f0653fc92e3ab4a (table test-table [id=d2ce78a01889448cbc387481fbb81610]) successfully deleted
I20260620 22:57:15.346735 9754 tablet_bootstrap.cc:492] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: No bootstrap required, opened a new log
I20260620 22:57:15.346804 9754 ts_tablet_manager.cc:1397] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 22:57:15.346949 9754 raft_consensus.cc:348] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.347010 9754 raft_consensus.cc:374] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.347028 9754 raft_consensus.cc:729] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183c090687e24c8b9fdb66563fd8dd7b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.347071 9754 consensus_queue.cc:260] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.347160 9754 ts_tablet_manager.cc:1428] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:57:15.354527 9760 raft_consensus.cc:482] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:57:15.354629 9760 raft_consensus.cc:504] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.354972 9760 leader_election.cc:288] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043), 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853)
I20260620 22:57:15.358456 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.355210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42853 (local address 127.1.166.3:46635)
0620 22:57:15.355373 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.355382 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.355453 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.355592 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.355592 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.355594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.355683 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.355685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.356611 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.356612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.357537 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.357540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.358116 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.358117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.358201 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.358205 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.358254 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.358304 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":55,"thread_start_us":69,"threads_started":1}
I20260620 22:57:15.358446 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.355271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:46635 (local address 127.1.166.2:42853)
0620 22:57:15.355315 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.355318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.355428 (+ 110us) server_negotiation.cc:407] Connection header received
0620 22:57:15.355535 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.355536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.355545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.355591 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.355764 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.356503 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.357636 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.358021 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.358085 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.358099 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.358266 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.358307 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:57:15.359011 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72da6a5d2bd04da18a0ce21ca77fcdfa" candidate_uuid: "ec2348b8cad04e40839875b035f6878b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183c090687e24c8b9fdb66563fd8dd7b" is_pre_election: true
I20260620 22:57:15.359123 9538 raft_consensus.cc:2454] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2348b8cad04e40839875b035f6878b in term 0.
I20260620 22:57:15.359251 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.355118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:43043 (local address 127.1.166.3:51351)
0620 22:57:15.355160 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.355171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.355224 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.355554 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.355555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.355557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.355655 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.355657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.356598 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.356599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.357852 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.357855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.358749 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.358750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.358925 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.358938 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.359060 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.359150 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":73}
I20260620 22:57:15.359251 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.355316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:51351 (local address 127.1.166.1:43043)
0620 22:57:15.355360 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.355376 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.355384 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:15.355411 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.355412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.355422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.355495 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.355729 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.356503 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.357978 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.358661 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.358739 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.358750 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.359093 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.359166 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:57:15.359355 9611 leader_election.cc:302] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c090687e24c8b9fdb66563fd8dd7b, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.359571 9760 raft_consensus.cc:2781] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:57:15.359623 9760 raft_consensus.cc:482] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:57:15.359656 9760 raft_consensus.cc:3037] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.360664 9760 raft_consensus.cc:504] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.360832 9760 leader_election.cc:288] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [CANDIDATE]: Term 1 election: Requested vote from peers 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043), 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853)
I20260620 22:57:15.361099 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72da6a5d2bd04da18a0ce21ca77fcdfa" candidate_uuid: "ec2348b8cad04e40839875b035f6878b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183c090687e24c8b9fdb66563fd8dd7b"
I20260620 22:57:15.361084 9398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72da6a5d2bd04da18a0ce21ca77fcdfa" candidate_uuid: "ec2348b8cad04e40839875b035f6878b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a5137ae25b454cb8fea7072f114575" is_pre_election: true
I20260620 22:57:15.361231 9538 raft_consensus.cc:3037] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.361255 9398 raft_consensus.cc:2454] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2348b8cad04e40839875b035f6878b in term 0.
I20260620 22:57:15.361383 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72da6a5d2bd04da18a0ce21ca77fcdfa" candidate_uuid: "ec2348b8cad04e40839875b035f6878b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a5137ae25b454cb8fea7072f114575"
I20260620 22:57:15.361431 9397 raft_consensus.cc:3037] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.362036 9538 raft_consensus.cc:2454] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2348b8cad04e40839875b035f6878b in term 1.
I20260620 22:57:15.362257 9397 raft_consensus.cc:2454] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2348b8cad04e40839875b035f6878b in term 1.
I20260620 22:57:15.362285 9611 leader_election.cc:302] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c090687e24c8b9fdb66563fd8dd7b, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.362859 9760 raft_consensus.cc:2781] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:15.363047 9760 raft_consensus.cc:686] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 1 LEADER]: Becoming Leader. State: Replica: ec2348b8cad04e40839875b035f6878b, State: Running, Role: LEADER
I20260620 22:57:15.363139 9760 consensus_queue.cc:237] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.364095 9239 catalog_manager.cc:5495] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec2348b8cad04e40839875b035f6878b (127.1.166.3). New cstate: current_term: 1 leader_uuid: "ec2348b8cad04e40839875b035f6878b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } health_report { overall_health: HEALTHY } } }
W20260620 22:57:15.391155 9725 tablet.cc:2369] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996234, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.62@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:57:15.402598 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.395167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.1.166.62:46797)
0620 22:57:15.395227 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.395229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.395299 (+ 70us) server_negotiation.cc:407] Connection header received
0620 22:57:15.395343 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.395344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.395355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.395438 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:15.395923 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.396693 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.397412 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.397696 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.397767 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.400107 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:15.400109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:15.400109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:15.400116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:15.401345 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.401799 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.401800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.401801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.401876 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.402078 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.402078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.402078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.402181 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:15.402236 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.402446 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.402528 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":29}
I20260620 22:57:15.402788 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.395080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46797 (local address 127.0.0.1:34988)
0620 22:57:15.395230 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.395240 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.395299 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.395474 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.395475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.395788 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:15.395842 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.395843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.396778 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.396779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.397326 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.397328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.397775 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.397776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.397825 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.398417 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:15.398422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:15.400005 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:15.401428 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:15.401435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:15.401437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:15.401708 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:15.401948 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:15.401949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:15.401950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:15.402021 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:15.402248 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:15.402256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:15.402311 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.402427 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.402612 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":40,"thread_start_us":84,"threads_started":1}
I20260620 22:57:15.405486 9239 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:34988:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:57:15.405768 9239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:57:15.407174 9378 tablet_service.cc:1467] Processing CreateTablet for tablet 00fcca91f253461b9f47fb8a042d6aa2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=722f167b859548d583c7a229084e78c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:57:15.407222 9518 tablet_service.cc:1467] Processing CreateTablet for tablet 00fcca91f253461b9f47fb8a042d6aa2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=722f167b859548d583c7a229084e78c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:57:15.407276 9658 tablet_service.cc:1467] Processing CreateTablet for tablet 00fcca91f253461b9f47fb8a042d6aa2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=722f167b859548d583c7a229084e78c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:57:15.407338 9378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fcca91f253461b9f47fb8a042d6aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.407338 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fcca91f253461b9f47fb8a042d6aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.407410 9658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fcca91f253461b9f47fb8a042d6aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.408792 9754 tablet_bootstrap.cc:492] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: Bootstrap starting.
I20260620 22:57:15.408792 9752 tablet_bootstrap.cc:492] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: Bootstrap starting.
I20260620 22:57:15.409168 9752 tablet_bootstrap.cc:654] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.409168 9754 tablet_bootstrap.cc:654] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.409579 9753 tablet_bootstrap.cc:492] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: Bootstrap starting.
I20260620 22:57:15.409996 9753 tablet_bootstrap.cc:654] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.410086 9754 tablet_bootstrap.cc:492] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: No bootstrap required, opened a new log
I20260620 22:57:15.410128 9754 ts_tablet_manager.cc:1397] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:15.410286 9754 raft_consensus.cc:348] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.410390 9754 raft_consensus.cc:374] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.410413 9754 raft_consensus.cc:729] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183c090687e24c8b9fdb66563fd8dd7b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.410449 9754 consensus_queue.cc:260] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.410539 9754 ts_tablet_manager.cc:1428] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.410614 9758 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.410719 9758 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:57:15.410759 9752 tablet_bootstrap.cc:492] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: No bootstrap required, opened a new log
I20260620 22:57:15.410804 9752 ts_tablet_manager.cc:1397] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:57:15.410948 9752 raft_consensus.cc:348] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.411013 9752 raft_consensus.cc:374] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.411032 9752 raft_consensus.cc:729] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a5137ae25b454cb8fea7072f114575, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.411070 9752 consensus_queue.cc:260] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.411156 9752 ts_tablet_manager.cc:1428] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.411283 9773 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.411360 9773 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:57:15.411860 9753 tablet_bootstrap.cc:492] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: No bootstrap required, opened a new log
I20260620 22:57:15.411913 9753 ts_tablet_manager.cc:1397] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:57:15.412074 9753 raft_consensus.cc:348] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.412137 9753 raft_consensus.cc:374] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.412154 9753 raft_consensus.cc:729] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2348b8cad04e40839875b035f6878b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.412195 9753 consensus_queue.cc:260] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.412284 9753 ts_tablet_manager.cc:1428] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.412338 9760 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.412431 9760 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:57:15.535244 9758 raft_consensus.cc:482] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:57:15.535356 9758 raft_consensus.cc:504] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.535666 9758 leader_election.cc:288] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043), ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511)
I20260620 22:57:15.540448 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.536130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:52631 (local address 127.1.166.1:43043)
0620 22:57:15.536202 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.536205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.536503 (+ 298us) server_negotiation.cc:407] Connection header received
0620 22:57:15.536531 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.536532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.536541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.536588 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.536794 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.537607 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.539385 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.539985 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.540075 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.540088 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.540317 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.540358 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:57:15.540448 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.536177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:42259 (local address 127.1.166.3:36511)
0620 22:57:15.536256 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.536258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.536435 (+ 177us) server_negotiation.cc:407] Connection header received
0620 22:57:15.536483 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.536485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.536497 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.536565 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:15.536794 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.537617 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.539385 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.539986 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.540075 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.540088 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.540317 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.540356 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:57:15.540514 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.536057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:36511 (local address 127.1.166.2:42259)
0620 22:57:15.536296 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.536311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.536420 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.536571 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.536571 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.536574 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.536683 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.536685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.537744 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.537746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.539249 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.539254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.540110 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.540111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.540219 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.540228 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.540306 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.540384 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":102,"thread_start_us":107,"threads_started":1}
I20260620 22:57:15.540510 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.536057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:43043 (local address 127.1.166.2:52631)
0620 22:57:15.536370 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.536379 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.536437 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.536596 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.536602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.536603 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:15.536683 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.536685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.537744 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.537746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.539249 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.539254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.540110 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.540111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.540219 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.540227 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.540306 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.540384 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":159,"thread_start_us":79,"threads_started":1}
I20260620 22:57:15.541195 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fcca91f253461b9f47fb8a042d6aa2" candidate_uuid: "183c090687e24c8b9fdb66563fd8dd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a5137ae25b454cb8fea7072f114575" is_pre_election: true
I20260620 22:57:15.541357 9397 raft_consensus.cc:2454] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183c090687e24c8b9fdb66563fd8dd7b in term 0.
I20260620 22:57:15.541446 9678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fcca91f253461b9f47fb8a042d6aa2" candidate_uuid: "183c090687e24c8b9fdb66563fd8dd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2348b8cad04e40839875b035f6878b" is_pre_election: true
I20260620 22:57:15.541580 9678 raft_consensus.cc:2454] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183c090687e24c8b9fdb66563fd8dd7b in term 0.
I20260620 22:57:15.541637 9471 leader_election.cc:302] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c090687e24c8b9fdb66563fd8dd7b, 90a5137ae25b454cb8fea7072f114575; no voters:
I20260620 22:57:15.541935 9758 raft_consensus.cc:2781] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:57:15.541977 9758 raft_consensus.cc:482] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:57:15.541991 9758 raft_consensus.cc:3037] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.542894 9758 raft_consensus.cc:504] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.543196 9758 leader_election.cc:288] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [CANDIDATE]: Term 1 election: Requested vote from peers 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043), ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511)
I20260620 22:57:15.543499 9678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fcca91f253461b9f47fb8a042d6aa2" candidate_uuid: "183c090687e24c8b9fdb66563fd8dd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2348b8cad04e40839875b035f6878b"
I20260620 22:57:15.543502 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fcca91f253461b9f47fb8a042d6aa2" candidate_uuid: "183c090687e24c8b9fdb66563fd8dd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a5137ae25b454cb8fea7072f114575"
I20260620 22:57:15.543596 9678 raft_consensus.cc:3037] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.543606 9397 raft_consensus.cc:3037] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.544344 9678 raft_consensus.cc:2454] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183c090687e24c8b9fdb66563fd8dd7b in term 1.
I20260620 22:57:15.544562 9397 raft_consensus.cc:2454] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183c090687e24c8b9fdb66563fd8dd7b in term 1.
I20260620 22:57:15.544605 9470 leader_election.cc:302] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c090687e24c8b9fdb66563fd8dd7b, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.544781 9758 raft_consensus.cc:2781] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:15.544957 9758 raft_consensus.cc:686] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 LEADER]: Becoming Leader. State: Replica: 183c090687e24c8b9fdb66563fd8dd7b, State: Running, Role: LEADER
I20260620 22:57:15.545046 9758 consensus_queue.cc:237] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.545478 9796 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: TxnStatusTablet state changed. Reason: New leader 183c090687e24c8b9fdb66563fd8dd7b. Latest consensus state: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.545588 9796 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:57:15.545868 9798 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:57:15.546022 9798 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:57:15.546049 9241 catalog_manager.cc:5495] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2). New cstate: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } health_report { overall_health: UNKNOWN } } }
I20260620 22:57:15.564284 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.560769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.1.166.2:42853)
0620 22:57:15.560848 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.560850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.560867 (+ 17us) server_negotiation.cc:407] Connection header received
0620 22:57:15.560913 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.560914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.560935 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.561028 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:15.561226 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.561746 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.563128 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.563358 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.563437 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.563718 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.563858 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.563949 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.564074 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.564098 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 22:57:15.564360 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.560641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42853 (local address 127.0.0.1:41560)
0620 22:57:15.560746 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.560759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.560810 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.561046 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.561047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.561049 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:57:15.561148 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.561152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.561833 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.561834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.563001 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.563005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.563448 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.563449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.563553 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.563580 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.563980 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.564007 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.564065 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.564174 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 22:57:15.566109 9678 raft_consensus.cc:1261] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Refusing update from remote peer 183c090687e24c8b9fdb66563fd8dd7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:15.566143 9397 raft_consensus.cc:1261] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Refusing update from remote peer 183c090687e24c8b9fdb66563fd8dd7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:15.566627 9758 consensus_queue.cc:1035] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:15.566830 9796 consensus_queue.cc:1035] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:15.567757 9773 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: TxnStatusTablet state changed. Reason: New leader 183c090687e24c8b9fdb66563fd8dd7b. Latest consensus state: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.567885 9773 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:57:15.567808 9760 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: TxnStatusTablet state changed. Reason: New leader 183c090687e24c8b9fdb66563fd8dd7b. Latest consensus state: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.567934 9760 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:57:15.569139 9796 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.569330 9796 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:57:15.569530 9773 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.569640 9773 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P 90a5137ae25b454cb8fea7072f114575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:57:15.569777 9766 mvcc.cc:205] Tried to move back new op lower bound from 7299056580875014144 to 7299056580793401344. Current Snapshot: MvccSnapshot[applied={T|T < 7299056580875014144 or (T in {7299056580875014144})}]
I20260620 22:57:15.570717 9758 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.570833 9758 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P 183c090687e24c8b9fdb66563fd8dd7b: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:57:15.571750 9760 tablet_replica.cc:439] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } } }
I20260620 22:57:15.571846 9760 tablet_replica.cc:442] T 00fcca91f253461b9f47fb8a042d6aa2 P ec2348b8cad04e40839875b035f6878b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:57:15.574308 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.571502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.1.166.3:36511)
0620 22:57:15.571543 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.571545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.571553 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:15.571590 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.571591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.571626 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.571701 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:15.571906 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.572556 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.573628 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.573817 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.573878 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.574022 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.574132 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.574180 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.574236 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.574253 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:57:15.579893 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.577295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:51179 (local address 127.1.166.2:42853)
0620 22:57:15.577351 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.577353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.577361 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:15.577383 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.577384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.577392 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.577438 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:15.577557 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.578031 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.579127 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.579335 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.579420 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.579659 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.579694 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.579748 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.579802 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.579827 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:57:15.579929 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.577194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42853 (local address 127.1.166.3:51179)
0620 22:57:15.577275 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.577285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.577320 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.577453 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.577454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.577455 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:57:15.577511 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.577512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.578092 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.578093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.579026 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.579029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.579455 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.579456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.579538 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.579559 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.579753 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.579755 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.579798 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.579848 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 22:57:15.584826 9397 raft_consensus.cc:1261] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Refusing update from remote peer ec2348b8cad04e40839875b035f6878b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:15.584832 9538 raft_consensus.cc:1261] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Refusing update from remote peer ec2348b8cad04e40839875b035f6878b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:15.585103 9783 consensus_queue.cc:1035] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [LEADER]: Connected to new peer: Peer: permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:15.585266 9760 consensus_queue.cc:1035] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:15.601902 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.598924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:55561 (local address 127.1.166.3:36511)
0620 22:57:15.598966 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.598968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.598976 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:15.599032 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.599034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.599045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.599096 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:15.599278 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.599763 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.601088 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.601335 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.601401 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.601595 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.601637 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.601681 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.601733 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.601763 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:57:15.601907 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.598691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:36511 (local address 127.1.166.2:55561)
0620 22:57:15.598908 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:15.598920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:15.598973 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:15.599100 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:15.599101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:15.599112 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:57:15.599187 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.599189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.599955 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.599986 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.601011 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:15.601015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.601408 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.601409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:15.601475 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.601511 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.601688 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.601689 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:57:15.601731 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:57:15.601791 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":104,"threads_started":1}
I20260620 22:57:15.613430 9241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:15.613579 9241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:15.614423 9241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: Sending DeleteTablet for 3 replicas of tablet 72da6a5d2bd04da18a0ce21ca77fcdfa
I20260620 22:57:15.614724 9378 tablet_service.cc:1514] Processing DeleteTablet for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38712
I20260620 22:57:15.614746 9658 tablet_service.cc:1514] Processing DeleteTablet for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:40444
I20260620 22:57:15.614817 9778 tablet_replica.cc:330] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: stopping tablet replica
I20260620 22:57:15.614847 9776 tablet_replica.cc:330] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: stopping tablet replica
I20260620 22:57:15.614866 9778 raft_consensus.cc:2229] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:15.614890 9776 raft_consensus.cc:2229] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:57:15.614904 9778 raft_consensus.cc:2258] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.614976 9776 raft_consensus.cc:2258] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.615321 9776 ts_tablet_manager.cc:1905] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.615512 9778 ts_tablet_manager.cc:1905] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.616983 9776 ts_tablet_manager.cc:1918] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:57:15.617038 9776 log.cc:1196] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/72da6a5d2bd04da18a0ce21ca77fcdfa
I20260620 22:57:15.617108 9778 ts_tablet_manager.cc:1918] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:57:15.617157 9778 log.cc:1196] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/72da6a5d2bd04da18a0ce21ca77fcdfa
I20260620 22:57:15.617254 9776 ts_tablet_manager.cc:1939] T 72da6a5d2bd04da18a0ce21ca77fcdfa P ec2348b8cad04e40839875b035f6878b: Deleting consensus metadata
I20260620 22:57:15.617384 9778 ts_tablet_manager.cc:1939] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 90a5137ae25b454cb8fea7072f114575: Deleting consensus metadata
I20260620 22:57:15.617605 9226 catalog_manager.cc:4841] TS ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511): tablet 72da6a5d2bd04da18a0ce21ca77fcdfa (table test-table [id=41e4388fa8a24c368f155c0d1a23a32e]) successfully deleted
I20260620 22:57:15.617687 9227 catalog_manager.cc:4841] TS 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043): tablet 72da6a5d2bd04da18a0ce21ca77fcdfa (table test-table [id=41e4388fa8a24c368f155c0d1a23a32e]) successfully deleted
I20260620 22:57:15.618247 9518 tablet_service.cc:1514] Processing DeleteTablet for tablet 72da6a5d2bd04da18a0ce21ca77fcdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:41554
I20260620 22:57:15.618377 9777 tablet_replica.cc:330] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: stopping tablet replica
I20260620 22:57:15.618428 9777 raft_consensus.cc:2229] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:15.618471 9777 raft_consensus.cc:2258] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.621141 9777 ts_tablet_manager.cc:1905] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.622877 9777 ts_tablet_manager.cc:1918] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:57:15.622936 9777 log.cc:1196] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/72da6a5d2bd04da18a0ce21ca77fcdfa
I20260620 22:57:15.623118 9777 ts_tablet_manager.cc:1939] T 72da6a5d2bd04da18a0ce21ca77fcdfa P 183c090687e24c8b9fdb66563fd8dd7b: Deleting consensus metadata
I20260620 22:57:15.624480 9227 catalog_manager.cc:4841] TS 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853): tablet 72da6a5d2bd04da18a0ce21ca77fcdfa (table test-table [id=41e4388fa8a24c368f155c0d1a23a32e]) successfully deleted
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:57:15.648216 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:15.640780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.1.166.62:46797)
0620 22:57:15.640837 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.640839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.640968 (+ 129us) server_negotiation.cc:407] Connection header received
0620 22:57:15.641030 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.641031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.641042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.641120 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:15.641870 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.642690 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.643764 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.644070 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.644143 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.648144 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34994: BlockingRecv error: recv got EOF from 127.0.0.1:34994 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 22:57:15 dist-test-slave-rdl9 krb5kdc[9196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM
I20260620 22:57:15.658634 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.649315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.1.166.62:46797)
0620 22:57:15.649358 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.649360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.649432 (+ 72us) server_negotiation.cc:407] Connection header received
0620 22:57:15.649503 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.649504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.649515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.649577 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:15.650158 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.650992 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.652237 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.652547 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.652621 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.655607 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:15.655609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:15.655609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:15.655619 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:15.657335 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.657786 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.657787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.657788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.657853 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:15.658054 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:15.658055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:15.658055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:15.658208 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:15.658307 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.658447 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.658533 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:57:15.660669 9241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:57:15.660866 9241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:57:15.662576 9658 tablet_service.cc:1467] Processing CreateTablet for tablet 2353bd040d7242b589c33cac05459a63 (DEFAULT_TABLE table=test-table [id=3260d56277b14cfc977be66f85a6a0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.662657 9518 tablet_service.cc:1467] Processing CreateTablet for tablet 2353bd040d7242b589c33cac05459a63 (DEFAULT_TABLE table=test-table [id=3260d56277b14cfc977be66f85a6a0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.662724 9658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2353bd040d7242b589c33cac05459a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.662858 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2353bd040d7242b589c33cac05459a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.664202 9378 tablet_service.cc:1467] Processing CreateTablet for tablet 2353bd040d7242b589c33cac05459a63 (DEFAULT_TABLE table=test-table [id=3260d56277b14cfc977be66f85a6a0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.664319 9378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2353bd040d7242b589c33cac05459a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.664736 9754 tablet_bootstrap.cc:492] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: Bootstrap starting.
I20260620 22:57:15.665176 9754 tablet_bootstrap.cc:654] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.665655 9753 tablet_bootstrap.cc:492] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: Bootstrap starting.
I20260620 22:57:15.666059 9753 tablet_bootstrap.cc:654] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.666209 9754 tablet_bootstrap.cc:492] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: No bootstrap required, opened a new log
I20260620 22:57:15.666267 9754 ts_tablet_manager.cc:1397] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:57:15.666406 9752 tablet_bootstrap.cc:492] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: Bootstrap starting.
I20260620 22:57:15.666466 9754 raft_consensus.cc:348] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.666540 9754 raft_consensus.cc:374] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.666559 9754 raft_consensus.cc:729] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183c090687e24c8b9fdb66563fd8dd7b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.666601 9754 consensus_queue.cc:260] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.666692 9754 ts_tablet_manager.cc:1428] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 22:57:15.666726 9752 tablet_bootstrap.cc:654] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.667985 9752 tablet_bootstrap.cc:492] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: No bootstrap required, opened a new log
I20260620 22:57:15.668040 9752 ts_tablet_manager.cc:1397] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:57:15.668154 9752 raft_consensus.cc:348] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.668226 9752 raft_consensus.cc:374] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.668246 9752 raft_consensus.cc:729] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a5137ae25b454cb8fea7072f114575, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.668288 9752 consensus_queue.cc:260] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.668393 9752 ts_tablet_manager.cc:1428] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.669351 9753 tablet_bootstrap.cc:492] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: No bootstrap required, opened a new log
I20260620 22:57:15.669409 9753 ts_tablet_manager.cc:1397] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 22:57:15.669565 9753 raft_consensus.cc:348] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.669639 9753 raft_consensus.cc:374] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.669658 9753 raft_consensus.cc:729] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2348b8cad04e40839875b035f6878b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.669700 9753 consensus_queue.cc:260] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.669785 9753 ts_tablet_manager.cc:1428] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:57:15.789354 9807 raft_consensus.cc:482] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:57:15.789496 9807 raft_consensus.cc:504] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.789675 9807 leader_election.cc:288] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043), 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853)
I20260620 22:57:15.789932 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2353bd040d7242b589c33cac05459a63" candidate_uuid: "ec2348b8cad04e40839875b035f6878b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a5137ae25b454cb8fea7072f114575" is_pre_election: true
I20260620 22:57:15.789923 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2353bd040d7242b589c33cac05459a63" candidate_uuid: "ec2348b8cad04e40839875b035f6878b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183c090687e24c8b9fdb66563fd8dd7b" is_pre_election: true
I20260620 22:57:15.790028 9397 raft_consensus.cc:2454] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2348b8cad04e40839875b035f6878b in term 0.
I20260620 22:57:15.790036 9538 raft_consensus.cc:2454] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2348b8cad04e40839875b035f6878b in term 0.
I20260620 22:57:15.790186 9611 leader_election.cc:302] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c090687e24c8b9fdb66563fd8dd7b, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.790299 9807 raft_consensus.cc:2781] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:57:15.790375 9807 raft_consensus.cc:482] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:57:15.790405 9807 raft_consensus.cc:3037] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.791268 9807 raft_consensus.cc:504] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.791430 9807 leader_election.cc:288] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [CANDIDATE]: Term 1 election: Requested vote from peers 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043), 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853)
I20260620 22:57:15.791760 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2353bd040d7242b589c33cac05459a63" candidate_uuid: "ec2348b8cad04e40839875b035f6878b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a5137ae25b454cb8fea7072f114575"
I20260620 22:57:15.791738 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2353bd040d7242b589c33cac05459a63" candidate_uuid: "ec2348b8cad04e40839875b035f6878b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183c090687e24c8b9fdb66563fd8dd7b"
I20260620 22:57:15.791896 9538 raft_consensus.cc:3037] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.791895 9397 raft_consensus.cc:3037] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.793213 9538 raft_consensus.cc:2454] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2348b8cad04e40839875b035f6878b in term 1.
I20260620 22:57:15.793213 9397 raft_consensus.cc:2454] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2348b8cad04e40839875b035f6878b in term 1.
I20260620 22:57:15.793589 9611 leader_election.cc:302] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c090687e24c8b9fdb66563fd8dd7b, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.793777 9807 raft_consensus.cc:2781] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:15.793862 9807 raft_consensus.cc:686] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 1 LEADER]: Becoming Leader. State: Replica: ec2348b8cad04e40839875b035f6878b, State: Running, Role: LEADER
I20260620 22:57:15.793924 9807 consensus_queue.cc:237] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.797537 9241 catalog_manager.cc:5495] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec2348b8cad04e40839875b035f6878b (127.1.166.3). New cstate: current_term: 1 leader_uuid: "ec2348b8cad04e40839875b035f6878b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } health_report { overall_health: HEALTHY } } }
I20260620 22:57:15.823503 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:15.821093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.1.166.3:36511)
0620 22:57:15.821230 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:15.821233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:15.821245 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:57:15.821276 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:15.821277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:15.821288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:15.821340 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:15.821495 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.821979 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.822858 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:15.823046 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:15.823113 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:15.823282 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:15.823310 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:15.823344 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:15.823394 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 22:57:15.823415 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 22:57:15.824362 9397 raft_consensus.cc:1261] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Refusing update from remote peer ec2348b8cad04e40839875b035f6878b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:15.824390 9538 raft_consensus.cc:1261] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Refusing update from remote peer ec2348b8cad04e40839875b035f6878b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:15.824625 9760 consensus_queue.cc:1035] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [LEADER]: Connected to new peer: Peer: permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:15.824726 9807 consensus_queue.cc:1035] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:15.827230 9239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:15.827316 9239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:15.827927 9239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: Sending DeleteTablet for 3 replicas of tablet 2353bd040d7242b589c33cac05459a63
I20260620 22:57:15.828145 9658 tablet_service.cc:1514] Processing DeleteTablet for tablet 2353bd040d7242b589c33cac05459a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:40444
I20260620 22:57:15.828178 9378 tablet_service.cc:1514] Processing DeleteTablet for tablet 2353bd040d7242b589c33cac05459a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38712
I20260620 22:57:15.828227 9776 tablet_replica.cc:330] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: stopping tablet replica
I20260620 22:57:15.828222 9518 tablet_service.cc:1514] Processing DeleteTablet for tablet 2353bd040d7242b589c33cac05459a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:15 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:41554
I20260620 22:57:15.828263 9778 tablet_replica.cc:330] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: stopping tablet replica
I20260620 22:57:15.828276 9776 raft_consensus.cc:2229] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:57:15.828305 9778 raft_consensus.cc:2229] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:15.828316 9777 tablet_replica.cc:330] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: stopping tablet replica
I20260620 22:57:15.828330 9776 raft_consensus.cc:2258] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.828341 9778 raft_consensus.cc:2258] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.828366 9777 raft_consensus.cc:2229] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:15.828408 9777 raft_consensus.cc:2258] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:15.828435 9239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:57:15.828563 9239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:57:15.828678 9777 ts_tablet_manager.cc:1905] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.828977 9778 ts_tablet_manager.cc:1905] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.829018 9776 ts_tablet_manager.cc:1905] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:15.829627 9518 tablet_service.cc:1467] Processing CreateTablet for tablet 3e96e17db9a54fc3924135b1188a0350 (DEFAULT_TABLE table=test-table [id=ff122777a0384124b4b51825046b5736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.829711 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e96e17db9a54fc3924135b1188a0350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.830192 9777 ts_tablet_manager.cc:1918] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:57:15.830240 9777 log.cc:1196] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/2353bd040d7242b589c33cac05459a63
I20260620 22:57:15.830380 9776 ts_tablet_manager.cc:1918] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:57:15.830432 9776 log.cc:1196] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/2353bd040d7242b589c33cac05459a63
I20260620 22:57:15.830467 9777 ts_tablet_manager.cc:1939] T 2353bd040d7242b589c33cac05459a63 P 183c090687e24c8b9fdb66563fd8dd7b: Deleting consensus metadata
I20260620 22:57:15.830467 9778 ts_tablet_manager.cc:1918] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:57:15.830510 9778 log.cc:1196] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/2353bd040d7242b589c33cac05459a63
I20260620 22:57:15.830623 9776 ts_tablet_manager.cc:1939] T 2353bd040d7242b589c33cac05459a63 P ec2348b8cad04e40839875b035f6878b: Deleting consensus metadata
I20260620 22:57:15.830742 9754 tablet_bootstrap.cc:492] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: Bootstrap starting.
I20260620 22:57:15.830806 9778 ts_tablet_manager.cc:1939] T 2353bd040d7242b589c33cac05459a63 P 90a5137ae25b454cb8fea7072f114575: Deleting consensus metadata
I20260620 22:57:15.830839 9227 catalog_manager.cc:4841] TS 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853): tablet 2353bd040d7242b589c33cac05459a63 (table test-table [id=3260d56277b14cfc977be66f85a6a0bf]) successfully deleted
I20260620 22:57:15.831032 9754 tablet_bootstrap.cc:654] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.831141 9227 catalog_manager.cc:4841] TS 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043): tablet 2353bd040d7242b589c33cac05459a63 (table test-table [id=3260d56277b14cfc977be66f85a6a0bf]) successfully deleted
I20260620 22:57:15.831166 9378 tablet_service.cc:1467] Processing CreateTablet for tablet 3e96e17db9a54fc3924135b1188a0350 (DEFAULT_TABLE table=test-table [id=ff122777a0384124b4b51825046b5736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.831261 9378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e96e17db9a54fc3924135b1188a0350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.831271 9226 catalog_manager.cc:4841] TS ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511): tablet 2353bd040d7242b589c33cac05459a63 (table test-table [id=3260d56277b14cfc977be66f85a6a0bf]) successfully deleted
I20260620 22:57:15.831349 9658 tablet_service.cc:1467] Processing CreateTablet for tablet 3e96e17db9a54fc3924135b1188a0350 (DEFAULT_TABLE table=test-table [id=ff122777a0384124b4b51825046b5736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:15.831430 9658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e96e17db9a54fc3924135b1188a0350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:15.832401 9752 tablet_bootstrap.cc:492] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575: Bootstrap starting.
I20260620 22:57:15.832536 9753 tablet_bootstrap.cc:492] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: Bootstrap starting.
I20260620 22:57:15.832762 9752 tablet_bootstrap.cc:654] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.832815 9753 tablet_bootstrap.cc:654] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:15.832818 9754 tablet_bootstrap.cc:492] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: No bootstrap required, opened a new log
I20260620 22:57:15.832868 9754 ts_tablet_manager.cc:1397] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:57:15.832990 9754 raft_consensus.cc:348] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.833067 9754 raft_consensus.cc:374] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.833091 9754 raft_consensus.cc:729] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183c090687e24c8b9fdb66563fd8dd7b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.833130 9754 consensus_queue.cc:260] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.833241 9754 ts_tablet_manager.cc:1428] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 22:57:15.833387 9753 tablet_bootstrap.cc:492] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: No bootstrap required, opened a new log
I20260620 22:57:15.833436 9753 ts_tablet_manager.cc:1397] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:15.833537 9753 raft_consensus.cc:348] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.833595 9753 raft_consensus.cc:374] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.833611 9753 raft_consensus.cc:729] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2348b8cad04e40839875b035f6878b, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.833649 9753 consensus_queue.cc:260] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.833750 9753 ts_tablet_manager.cc:1428] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.833962 9752 tablet_bootstrap.cc:492] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575: No bootstrap required, opened a new log
I20260620 22:57:15.834010 9752 ts_tablet_manager.cc:1397] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260620 22:57:15.834118 9752 raft_consensus.cc:348] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.834180 9752 raft_consensus.cc:374] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:15.834197 9752 raft_consensus.cc:729] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a5137ae25b454cb8fea7072f114575, State: Initialized, Role: FOLLOWER
I20260620 22:57:15.834239 9752 consensus_queue.cc:260] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.834322 9752 ts_tablet_manager.cc:1428] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:15.983199 9811 raft_consensus.cc:482] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:57:15.983316 9811 raft_consensus.cc:504] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.983479 9811 leader_election.cc:288] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043), ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511)
I20260620 22:57:15.983737 9678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e96e17db9a54fc3924135b1188a0350" candidate_uuid: "183c090687e24c8b9fdb66563fd8dd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2348b8cad04e40839875b035f6878b" is_pre_election: true
I20260620 22:57:15.983739 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e96e17db9a54fc3924135b1188a0350" candidate_uuid: "183c090687e24c8b9fdb66563fd8dd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a5137ae25b454cb8fea7072f114575" is_pre_election: true
I20260620 22:57:15.983846 9678 raft_consensus.cc:2454] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183c090687e24c8b9fdb66563fd8dd7b in term 0.
I20260620 22:57:15.983860 9397 raft_consensus.cc:2454] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183c090687e24c8b9fdb66563fd8dd7b in term 0.
I20260620 22:57:15.984030 9470 leader_election.cc:302] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c090687e24c8b9fdb66563fd8dd7b, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.984130 9811 raft_consensus.cc:2781] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:57:15.984189 9811 raft_consensus.cc:482] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:57:15.984202 9811 raft_consensus.cc:3037] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.984943 9811 raft_consensus.cc:504] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.985078 9811 leader_election.cc:288] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [CANDIDATE]: Term 1 election: Requested vote from peers 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043), ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511)
I20260620 22:57:15.985282 9397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e96e17db9a54fc3924135b1188a0350" candidate_uuid: "183c090687e24c8b9fdb66563fd8dd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a5137ae25b454cb8fea7072f114575"
I20260620 22:57:15.985291 9678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e96e17db9a54fc3924135b1188a0350" candidate_uuid: "183c090687e24c8b9fdb66563fd8dd7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2348b8cad04e40839875b035f6878b"
I20260620 22:57:15.985364 9678 raft_consensus.cc:3037] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.985366 9397 raft_consensus.cc:3037] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:15.985970 9678 raft_consensus.cc:2454] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183c090687e24c8b9fdb66563fd8dd7b in term 1.
I20260620 22:57:15.986059 9397 raft_consensus.cc:2454] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183c090687e24c8b9fdb66563fd8dd7b in term 1.
I20260620 22:57:15.986171 9470 leader_election.cc:302] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183c090687e24c8b9fdb66563fd8dd7b, ec2348b8cad04e40839875b035f6878b; no voters:
I20260620 22:57:15.986272 9811 raft_consensus.cc:2781] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:15.986364 9811 raft_consensus.cc:686] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 LEADER]: Becoming Leader. State: Replica: 183c090687e24c8b9fdb66563fd8dd7b, State: Running, Role: LEADER
I20260620 22:57:15.986451 9811 consensus_queue.cc:237] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } }
I20260620 22:57:15.987068 9241 catalog_manager.cc:5495] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2). New cstate: current_term: 1 leader_uuid: "183c090687e24c8b9fdb66563fd8dd7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183c090687e24c8b9fdb66563fd8dd7b" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 } health_report { overall_health: UNKNOWN } } }
I20260620 22:57:16.023886 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:16.021582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.1.166.2:42853)
0620 22:57:16.021626 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:16.021628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:16.021642 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:57:16.021707 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:16.021708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:16.021717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:16.021762 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:16.021851 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:16.022499 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:16.023258 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:16.023463 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:16.023522 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:16.023680 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:16.023709 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:16.023742 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:16.023783 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 22:57:16.023803 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:57:16.028534 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:16.026483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42853 (local address 127.1.166.2:55171)
0620 22:57:16.026577 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:16.026589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:16.026642 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:16.026754 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:16.026755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:16.026756 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:57:16.026804 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:16.026805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:16.027370 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:16.027371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:16.027914 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:16.027915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:16.028200 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:16.028201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:16.028251 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:16.028255 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:57:16.028381 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:57:16.028382 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:57:16.028421 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 22:57:16.028465 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":2}
I20260620 22:57:16.028534 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:16.026587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:55171 (local address 127.1.166.2:42853)
0620 22:57:16.026619 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:16.026621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:16.026629 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:16.026695 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:16.026696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:16.026705 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:16.026748 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:16.026859 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:16.027306 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:16.027981 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:16.028148 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:16.028198 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:16.028307 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:16.028337 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:16.028377 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:16.028422 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 22:57:16.028446 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":15}
I20260620 22:57:16.031267 9397 raft_consensus.cc:1261] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Refusing update from remote peer 183c090687e24c8b9fdb66563fd8dd7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:16.031276 9678 raft_consensus.cc:1261] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Refusing update from remote peer 183c090687e24c8b9fdb66563fd8dd7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:16.031539 9796 consensus_queue.cc:1035] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a5137ae25b454cb8fea7072f114575" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:16.031699 9758 consensus_queue.cc:1035] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec2348b8cad04e40839875b035f6878b" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:16.043385 9241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:16.043534 9241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:16.044337 9241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b45f012cf44c498a90299bb140d52a4a: Sending DeleteTablet for 3 replicas of tablet 3e96e17db9a54fc3924135b1188a0350
I20260620 22:57:16.044663 9658 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e96e17db9a54fc3924135b1188a0350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:16 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:40444
I20260620 22:57:16.044691 9518 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e96e17db9a54fc3924135b1188a0350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:16 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:41554
I20260620 22:57:16.044692 9378 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e96e17db9a54fc3924135b1188a0350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:16 UTC) from {username='oryx', principal='oryx/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38712
I20260620 22:57:16.044795 9777 tablet_replica.cc:330] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: stopping tablet replica
I20260620 22:57:16.044795 9776 tablet_replica.cc:330] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: stopping tablet replica
I20260620 22:57:16.044854 9776 raft_consensus.cc:2229] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:16.044852 9777 raft_consensus.cc:2229] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:57:16.044900 9776 raft_consensus.cc:2258] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:16.044981 9777 raft_consensus.cc:2258] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:16.045076 9778 tablet_replica.cc:330] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575: stopping tablet replica
I20260620 22:57:16.045172 9778 raft_consensus.cc:2229] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:16.045230 9778 raft_consensus.cc:2258] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:16.045390 9776 ts_tablet_manager.cc:1905] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:16.045686 9777 ts_tablet_manager.cc:1905] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:16.046979 9776 ts_tablet_manager.cc:1918] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:57:16.047044 9776 log.cc:1196] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/3e96e17db9a54fc3924135b1188a0350
I20260620 22:57:16.047240 9776 ts_tablet_manager.cc:1939] T 3e96e17db9a54fc3924135b1188a0350 P ec2348b8cad04e40839875b035f6878b: Deleting consensus metadata
I20260620 22:57:16.047457 9777 ts_tablet_manager.cc:1918] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:57:16.047521 9777 log.cc:1196] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/3e96e17db9a54fc3924135b1188a0350
I20260620 22:57:16.047685 9226 catalog_manager.cc:4841] TS ec2348b8cad04e40839875b035f6878b (127.1.166.3:36511): tablet 3e96e17db9a54fc3924135b1188a0350 (table test-table [id=ff122777a0384124b4b51825046b5736]) successfully deleted
I20260620 22:57:16.047719 9777 ts_tablet_manager.cc:1939] T 3e96e17db9a54fc3924135b1188a0350 P 183c090687e24c8b9fdb66563fd8dd7b: Deleting consensus metadata
I20260620 22:57:16.048128 9227 catalog_manager.cc:4841] TS 183c090687e24c8b9fdb66563fd8dd7b (127.1.166.2:42853): tablet 3e96e17db9a54fc3924135b1188a0350 (table test-table [id=ff122777a0384124b4b51825046b5736]) successfully deleted
I20260620 22:57:16.048199 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 9313
I20260620 22:57:16.048403 9778 ts_tablet_manager.cc:1905] T 3e96e17db9a54fc3924135b1188a0350 P 90a5137ae25b454cb8fea7072f114575: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 22:57:16.055449 9227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:43043 (error 108)
I20260620 22:57:16.055814 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 9453
I20260620 22:57:16.056597 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:16.055886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:43043 (local address 127.0.0.1:38720)
0620 22:57:16.056500 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:16.056553 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:43043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":44,"threads_started":1}
W20260620 22:57:16.056697 9227 catalog_manager.cc:4584] TS 90a5137ae25b454cb8fea7072f114575 (127.1.166.1:43043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e96e17db9a54fc3924135b1188a0350: Network error: Client connection negotiation failed: client connection to 127.1.166.1:43043: connect: Connection refused (error 111)
I20260620 22:57:16.062275 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 9593
I20260620 22:57:16.068148 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 9209
2026-06-20T22:57:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2761 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:57:16 dist-test-slave-rdl9 krb5kdc[9844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:57:16 dist-test-slave-rdl9 krb5kdc[9844](info): set up 2 sockets
Jun 20 22:57:16 dist-test-slave-rdl9 krb5kdc[9844](info): commencing operation
krb5kdc: starting...
W20260620 22:57:17.059003 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:57:17 dist-test-slave-rdl9 krb5kdc[9844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:57:17.185106 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34663
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:34663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:17.271044 9857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:17.271293 9857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:17.271327 9857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:17.272742 9857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:57:17.272818 9857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:17.272840 9857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:17.272855 9857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:57:17.272871 9857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:57:17.274859 9857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:34663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:17.275182 9857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:17.275414 9857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:17.280510 9862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:17.280543 9863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:17.280759 9857 server_base.cc:1034] running on GCE node
W20260620 22:57:17.280510 9865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:17.281093 9857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:17.281404 9857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:17.282605 9857 hybrid_clock.cc:648] HybridClock initialized: now 1781996237282584 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T22:57:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1224 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:57:17 dist-test-slave-rdl9 krb5kdc[9870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:57:17 dist-test-slave-rdl9 krb5kdc[9870](info): set up 2 sockets
Jun 20 22:57:17 dist-test-slave-rdl9 krb5kdc[9870](info): commencing operation
krb5kdc: starting...
W20260620 22:57:18.286806 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:57:18 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:57:18.417001 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38171
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:18.509804 9883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:18.510066 9883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:18.510093 9883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:18.512023 9883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:57:18.512127 9883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:18.512153 9883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:18.512176 9883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:57:18.512195 9883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:57:18.514518 9883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:18.514849 9883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:18.515115 9883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:18.521184 9888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:18.521219 9889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:18.521196 9883 server_base.cc:1034] running on GCE node
W20260620 22:57:18.521425 9891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:18.522032 9883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:18.522508 9883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:18.523691 9883 hybrid_clock.cc:648] HybridClock initialized: now 1781996238523653 us; error 61 us; skew 500 ppm
Jun 20 22:57:18 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996238, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:18.531749 9883 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:57:18.532439 9883 webserver.cc:460] Webserver started at http://127.1.166.62:43087/ using document root <none> and password file <none>
I20260620 22:57:18.532713 9883 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:18.532768 9883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:18.533083 9883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:18.534602 9883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "230af62daf56462fa018882d94838b06"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "57694d92e4d96d9cdea4dc46c417f7cd"
server_key_iv: "591005da99a59460108b4e6400005860"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.535171 9883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "230af62daf56462fa018882d94838b06"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "57694d92e4d96d9cdea4dc46c417f7cd"
server_key_iv: "591005da99a59460108b4e6400005860"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.536999 9883 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 22:57:18.538090 9898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:18.538374 9883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:18.538479 9883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "230af62daf56462fa018882d94838b06"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "57694d92e4d96d9cdea4dc46c417f7cd"
server_key_iv: "591005da99a59460108b4e6400005860"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.538590 9883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:18.569068 9883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:18.571640 9883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:18.571803 9883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:18.576229 9883 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:38171
I20260620 22:57:18.576357 9950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:38171 every 8 connection(s)
I20260620 22:57:18.576812 9883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:57:18.577958 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:18.580570 9951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Bootstrap starting.
I20260620 22:57:18.581282 9951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:18.581578 9951 log.cc:824] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:18.582598 9951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: No bootstrap required, opened a new log
I20260620 22:57:18.583784 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 9883
I20260620 22:57:18.583886 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:57:18.584077 1688 external_mini_cluster.cc:1428] Setting key 7d4367b8cef347b6f48ef66cee3ddde7
I20260620 22:57:18.584506 9951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } }
I20260620 22:57:18.584730 9951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:18.584772 9951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230af62daf56462fa018882d94838b06, State: Initialized, Role: FOLLOWER
I20260620 22:57:18.584918 9951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } }
I20260620 22:57:18.585000 9951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:57:18.585024 9951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:57:18.585074 9951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:18.586277 9951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } }
I20260620 22:57:18.586416 9951 leader_election.cc:302] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230af62daf56462fa018882d94838b06; no voters:
I20260620 22:57:18.586611 9951 leader_election.cc:288] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:57:18.586663 9956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:18.586798 9956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 1 LEADER]: Becoming Leader. State: Replica: 230af62daf56462fa018882d94838b06, State: Running, Role: LEADER
I20260620 22:57:18.586890 9951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:57:18.586881 9956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } }
I20260620 22:57:18.587227 9956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230af62daf56462fa018882d94838b06. Latest consensus state: current_term: 1 leader_uuid: "230af62daf56462fa018882d94838b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } } }
I20260620 22:57:18.587303 9956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:18.587448 9958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "230af62daf56462fa018882d94838b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } } }
I20260620 22:57:18.587519 9958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:18.587651 9962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:57:18.588471 9962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:57:18.590723 9962 catalog_manager.cc:1347] Generated new cluster ID: f9ee144c042c4e6595d9f6a915656755
I20260620 22:57:18.590790 9962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:57:18 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:57:18.596033 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:18.584651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.1.166.62:38171)
0620 22:57:18.584840 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:18.584851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:18.584888 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:57:18.585024 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:18.585032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:18.585108 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:18.585274 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:18.585994 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:18.587241 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:18.588326 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:18.588950 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:18.589065 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:18.591820 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:18.591826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:18.591827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:18.591842 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:18.594011 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:18.594531 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:18.594536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:18.594537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:18.594702 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:18.594913 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:18.594914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:18.594915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:18.595303 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:18.595512 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:18.595599 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 22:57:18.595777 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260620 22:57:18.610025 9962 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:57:18.610698 9962 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:57:18.617537 9962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Generated new TSK 0
I20260620 22:57:18.617812 9962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:57:18.655791 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38171
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:18.780001 9977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:18.780252 9977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:18.780279 9977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:18.781555 9977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:18.781607 9977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:18.781636 9977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:57:18.783205 9977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:18.783438 9977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:18.783640 9977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:18.788614 9983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:18.788686 9982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:18.788935 9985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:18.789328 9977 server_base.cc:1034] running on GCE node
I20260620 22:57:18.789589 9977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:18.789852 9977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:18.791025 9977 hybrid_clock.cc:648] HybridClock initialized: now 1781996238791008 us; error 37 us; skew 500 ppm
Jun 20 22:57:18 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996238, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:18.794763 9977 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:57:18.795423 9977 webserver.cc:460] Webserver started at http://127.1.166.1:42025/ using document root <none> and password file <none>
I20260620 22:57:18.795730 9977 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:18.795791 9977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:18.795979 9977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:18.797437 9977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "4310d0b60ee14de39b19a19f9c842c60"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "8ac40979662b9e0fcdd0543c4d294c39"
server_key_iv: "a9a39ca66a36621134be3949660a5abc"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.797880 9977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "4310d0b60ee14de39b19a19f9c842c60"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "8ac40979662b9e0fcdd0543c4d294c39"
server_key_iv: "a9a39ca66a36621134be3949660a5abc"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.799391 9977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 22:57:18.800530 9992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:18.800721 9977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:18.800815 9977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "4310d0b60ee14de39b19a19f9c842c60"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "8ac40979662b9e0fcdd0543c4d294c39"
server_key_iv: "a9a39ca66a36621134be3949660a5abc"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.800890 9977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:18.814543 9977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:18.817028 9977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:18.817198 9977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:18.817428 9977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:18.817747 9977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:18.817821 9977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:18.817874 9977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:18.817905 9977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:18.823908 9977 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:42961
I20260620 22:57:18.823957 10105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:42961 every 8 connection(s)
I20260620 22:57:18.824347 9977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:57:18 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996238, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:57:18.833792 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 9977
I20260620 22:57:18.833901 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:57:18.834080 1688 external_mini_cluster.cc:1428] Setting key a0ee23534c01b425e7fa7e1667036613
I20260620 22:57:18.835687 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:18.825213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:38795 (local address 127.1.166.62:38171)
0620 22:57:18.825301 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:18.825303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:18.825398 (+ 95us) server_negotiation.cc:407] Connection header received
0620 22:57:18.826019 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:18.826021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:18.826047 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:18.826150 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:18.827269 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:18.827826 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:18.829065 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:18.829334 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:18.829416 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:18.832263 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:18.832265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:18.832266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:18.832280 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:18.833653 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:18.834181 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:18.834183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:18.834183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:18.834248 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:18.834603 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:18.834604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:18.834604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:18.834754 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:18.834815 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:18.835383 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 22:57:18.835555 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 22:57:18.835923 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:18.825115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:38795)
0620 22:57:18.825279 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:18.825318 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:18.825854 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:18.826225 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:18.826229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:18.826872 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:18.827107 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:18.827112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:18.827994 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:18.827995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:18.828913 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:18.828916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:18.829464 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:18.829465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:18.829616 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:18.830225 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:18.830235 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:18.832098 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:18.833764 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:18.833776 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:18.833779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:18.834088 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:18.834325 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:18.834326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:18.834326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:18.834527 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:18.834916 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:18.834934 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:18.835183 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 22:57:18.835315 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 22:57:18.835560 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260620 22:57:18.836577 10106 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38171
I20260620 22:57:18.836733 10106 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:18.836954 10106 heartbeater.cc:511] Master 127.1.166.62:38171 requested a full tablet report, sending...
I20260620 22:57:18.837668 9915 ts_manager.cc:194] Registered new tserver with Master: 4310d0b60ee14de39b19a19f9c842c60 (127.1.166.1:42961)
I20260620 22:57:18.838388 9915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:38795
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:57:18.861337 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38171
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:18.947189 10111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:18.947412 10111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:18.947444 10111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:18.948683 10111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:18.948734 10111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:18.948772 10111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:57:18.950284 10111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:18.950572 10111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:18.950814 10111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:18.956090 10117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:18.956090 10119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:18.956126 10116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:18.956741 10111 server_base.cc:1034] running on GCE node
I20260620 22:57:18.957077 10111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:18.957352 10111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:18.958499 10111 hybrid_clock.cc:648] HybridClock initialized: now 1781996238958482 us; error 38 us; skew 500 ppm
Jun 20 22:57:18 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996238, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:18.961701 10111 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:57:18.962258 10111 webserver.cc:460] Webserver started at http://127.1.166.2:43411/ using document root <none> and password file <none>
I20260620 22:57:18.962499 10111 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:18.962549 10111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:18.962680 10111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:18.963749 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "37b0d415d78b4898a31b11e8b3cdf2f9"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "bf98940f62ac0c125c66bfeb4d57d93c"
server_key_iv: "028afde5361cd1761770dbdfb81daf8c"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.964085 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "37b0d415d78b4898a31b11e8b3cdf2f9"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "bf98940f62ac0c125c66bfeb4d57d93c"
server_key_iv: "028afde5361cd1761770dbdfb81daf8c"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.965402 10111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:57:18.966281 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:18.966445 10111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:18.966532 10111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "37b0d415d78b4898a31b11e8b3cdf2f9"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "bf98940f62ac0c125c66bfeb4d57d93c"
server_key_iv: "028afde5361cd1761770dbdfb81daf8c"
server_key_version: "encryptionkey@0"
I20260620 22:57:18.966598 10111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:19.014694 10111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:19.017058 10111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:19.017186 10111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:19.017401 10111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:19.017727 10111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:19.017769 10111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.017793 10111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:19.017805 10111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.024006 10111 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:42591
I20260620 22:57:19.024053 10239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:42591 every 8 connection(s)
I20260620 22:57:19.024416 10111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:57:19.027385 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 10111
I20260620 22:57:19.027508 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:57:19.027731 1688 external_mini_cluster.cc:1428] Setting key 95b2be2548862638764c95c1677df316
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996238, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
I20260620 22:57:19.035691 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.025211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:58633 (local address 127.1.166.62:38171)
0620 22:57:19.025298 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.025300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.025501 (+ 201us) server_negotiation.cc:407] Connection header received
0620 22:57:19.026081 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.026084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.026103 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.026212 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.027515 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.028247 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.029484 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.029786 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.029891 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.032775 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:19.032777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:19.032777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:19.032788 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:19.034141 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:19.034669 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:19.034670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:19.034671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:19.034728 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:19.034985 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:19.034986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:19.034987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:19.035111 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:19.035168 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:19.035481 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 22:57:19.035589 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 22:57:19.035881 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.025172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:58633)
0620 22:57:19.025362 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.025409 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.025914 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.026375 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.026380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.027045 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.027333 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.027340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.028362 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.028363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.029353 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.029357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.029939 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.029940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.030139 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.030846 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.030860 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.032621 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:19.034249 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:19.034261 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:19.034265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:19.034593 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:19.034790 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:19.034791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:19.034792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:19.034915 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:19.035173 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:19.035180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:19.035341 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 22:57:19.035435 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:57:19.035616 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260620 22:57:19.036494 10240 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38171
I20260620 22:57:19.036613 10240 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:19.036810 10240 heartbeater.cc:511] Master 127.1.166.62:38171 requested a full tablet report, sending...
I20260620 22:57:19.037308 9915 ts_manager.cc:194] Registered new tserver with Master: 37b0d415d78b4898a31b11e8b3cdf2f9 (127.1.166.2:42591)
I20260620 22:57:19.037667 9915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:58633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:57:19.051414 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38171
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:19.142956 10245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:19.143198 10245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:19.143229 10245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:19.144719 10245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:19.144804 10245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:19.144840 10245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:57:19.146565 10245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:19.146857 10245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:19.147128 10245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:19.152438 10253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.152431 10250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.152597 10251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:19.152998 10245 server_base.cc:1034] running on GCE node
I20260620 22:57:19.153285 10245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:19.153571 10245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:19.154744 10245 hybrid_clock.cc:648] HybridClock initialized: now 1781996239154714 us; error 39 us; skew 500 ppm
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996239, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:19.157879 10245 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:57:19.158413 10245 webserver.cc:460] Webserver started at http://127.1.166.3:38197/ using document root <none> and password file <none>
I20260620 22:57:19.158656 10245 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:19.158710 10245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:19.158854 10245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:19.159947 10245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "7d81376b7faf4d10a18f718113cbd00e"
format_stamp: "Formatted at 2026-06-20 22:57:19 on dist-test-slave-rdl9"
server_key: "9c89ebf286c07e65453b7ea4953f1c73"
server_key_iv: "23fe34a8e0532088768553c62cd08582"
server_key_version: "encryptionkey@0"
I20260620 22:57:19.160295 10245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "7d81376b7faf4d10a18f718113cbd00e"
format_stamp: "Formatted at 2026-06-20 22:57:19 on dist-test-slave-rdl9"
server_key: "9c89ebf286c07e65453b7ea4953f1c73"
server_key_iv: "23fe34a8e0532088768553c62cd08582"
server_key_version: "encryptionkey@0"
I20260620 22:57:19.161715 10245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:57:19.162623 10260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.162820 10245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:19.162914 10245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "7d81376b7faf4d10a18f718113cbd00e"
format_stamp: "Formatted at 2026-06-20 22:57:19 on dist-test-slave-rdl9"
server_key: "9c89ebf286c07e65453b7ea4953f1c73"
server_key_iv: "23fe34a8e0532088768553c62cd08582"
server_key_version: "encryptionkey@0"
I20260620 22:57:19.162987 10245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:19.183718 10245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:19.186684 10245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:19.186857 10245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:19.187139 10245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:19.187516 10245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:19.187562 10245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.187597 10245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:19.187613 10245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.194327 10245 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:36073
I20260620 22:57:19.194546 10373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:36073 every 8 connection(s)
I20260620 22:57:19.194834 10245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:57:19.197980 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 10245
I20260620 22:57:19.198098 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:57:19.198283 1688 external_mini_cluster.cc:1428] Setting key b6a3c1d8acea544f6f11548ebf153659
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996239, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:57:19.205175 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.195680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:51205 (local address 127.1.166.62:38171)
0620 22:57:19.195745 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.195747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.196011 (+ 264us) server_negotiation.cc:407] Connection header received
0620 22:57:19.196490 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.196491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.196504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.196698 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.198149 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.198788 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.199904 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.200110 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.200168 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.202532 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:19.202533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:19.202533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:19.202542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:19.203796 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:19.204194 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:19.204195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:19.204195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:19.204256 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:19.204476 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:19.204476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:19.204477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:19.204635 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:19.204692 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:19.204994 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 22:57:19.205082 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:57:19.205377 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.195636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:51205)
0620 22:57:19.195816 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.195863 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.196403 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.196849 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.196855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.197744 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.197983 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.197992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.198880 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.198881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.199799 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.199802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.200188 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.200189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.200322 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.200855 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.200865 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.202432 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:19.203871 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:19.203880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:19.203883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:19.204121 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:19.204314 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:19.204315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:19.204315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:19.204418 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:19.204690 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:19.204697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:19.204868 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 22:57:19.204963 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 22:57:19.205144 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260620 22:57:19.206284 10374 heartbeater.cc:347] Connected to a master server at 127.1.166.62:38171
I20260620 22:57:19.206555 10374 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:19.206852 10374 heartbeater.cc:511] Master 127.1.166.62:38171 requested a full tablet report, sending...
I20260620 22:57:19.207355 9915 ts_manager.cc:194] Registered new tserver with Master: 7d81376b7faf4d10a18f718113cbd00e (127.1.166.3:36073)
I20260620 22:57:19.207749 9915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:51205
I20260620 22:57:19.210474 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab.
Entry for principal oryx/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab.
WARNING: no policy specified for oryx/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab.
Entry for principal oryx/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab.
WARNING: no policy specified for oryx/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab.
Entry for principal oryx/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab.
I20260620 22:57:19.288892 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 9977
I20260620 22:57:19.293032 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 10111
I20260620 22:57:19.297379 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 10245
I20260620 22:57:19.302510 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 9883
I20260620 22:57:19.307941 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38171
--webserver_interface=127.1.166.62
--webserver_port=43087
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:19.398458 10383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:19.398730 10383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:19.398782 10383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:19.400403 10383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:57:19.400486 10383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:19.400508 10383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:19.400524 10383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:57:19.400542 10383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:57:19.402426 10383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=43087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:19.402702 10383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:19.402964 10383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:19.407872 10389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.407892 10388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.407946 10391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:19.408249 10383 server_base.cc:1034] running on GCE node
I20260620 22:57:19.408489 10383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:19.408818 10383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:19.410007 10383 hybrid_clock.cc:648] HybridClock initialized: now 1781996239409982 us; error 38 us; skew 500 ppm
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996239, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:19.413575 10383 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:57:19.414466 10383 webserver.cc:460] Webserver started at http://127.1.166.62:43087/ using document root <none> and password file <none>
I20260620 22:57:19.414727 10383 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:19.414798 10383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:19.416688 10383 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:57:19.417829 10398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.418164 10383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:19.418290 10383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "230af62daf56462fa018882d94838b06"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "57694d92e4d96d9cdea4dc46c417f7cd"
server_key_iv: "591005da99a59460108b4e6400005860"
server_key_version: "encryptionkey@0"
I20260620 22:57:19.418808 10383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:19.437587 10383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:19.440325 10383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:19.440516 10383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:19.444957 10383 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:38171
I20260620 22:57:19.445047 10450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:38171 every 8 connection(s)
I20260620 22:57:19.445582 10383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:57:19.446979 10451 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:57:19.447666 10451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Bootstrap starting.
I20260620 22:57:19.453874 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 10383
I20260620 22:57:19.454236 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:42961
--local_ip_for_outbound_sockets=127.1.166.1
--tserver_master_addrs=127.1.166.62:38171
--webserver_port=42025
--webserver_interface=127.1.166.1
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.166.1
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:57:19.456425 10451 log.cc:824] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:19.458956 10451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:57:19.459303 10451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Bootstrap complete.
I20260620 22:57:19.462690 10451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } }
I20260620 22:57:19.463090 10451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230af62daf56462fa018882d94838b06, State: Initialized, Role: FOLLOWER
I20260620 22:57:19.463284 10451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } }
I20260620 22:57:19.463372 10451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:57:19.463426 10451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:57:19.463481 10451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:57:19.465175 10451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } }
I20260620 22:57:19.465335 10451 leader_election.cc:302] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230af62daf56462fa018882d94838b06; no voters:
I20260620 22:57:19.465557 10451 leader_election.cc:288] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:57:19.465659 10455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:57:19.465898 10451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:57:19.465930 10455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [term 2 LEADER]: Becoming Leader. State: Replica: 230af62daf56462fa018882d94838b06, State: Running, Role: LEADER
I20260620 22:57:19.466084 10455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } }
I20260620 22:57:19.466634 10455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230af62daf56462fa018882d94838b06. Latest consensus state: current_term: 2 leader_uuid: "230af62daf56462fa018882d94838b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } } }
I20260620 22:57:19.466784 10455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:19.466971 10457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "230af62daf56462fa018882d94838b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230af62daf56462fa018882d94838b06" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 38171 } } }
I20260620 22:57:19.467042 10457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06 [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:19.467195 10464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:57:19.467801 10464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:57:19.467964 10464 catalog_manager.cc:1259] Loaded cluster ID: f9ee144c042c4e6595d9f6a915656755
I20260620 22:57:19.468005 10464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:57:19.468930 10464 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:57:19.469249 10464 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 230af62daf56462fa018882d94838b06: Loaded TSK: 0
I20260620 22:57:19.469478 10464 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:57:19.562289 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:19.562585 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:19.562625 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:19.564008 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:19.564077 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:19.564121 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:57:19.566490 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.1.keytab
--principal=oryx/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:42961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=42025
--tserver_master_addrs=127.1.166.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:19.566846 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:19.567147 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:19.572453 10480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.572454 10477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.572811 10478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:19.573582 10453 server_base.cc:1034] running on GCE node
I20260620 22:57:19.573841 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:19.574115 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:19.575301 10453 hybrid_clock.cc:648] HybridClock initialized: now 1781996239575244 us; error 76 us; skew 500 ppm
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996239, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:19.577980 10453 init.cc:377] Logged in from keytab as oryx/127.1.166.1@KRBTEST.COM (short username oryx)
I20260620 22:57:19.578557 10453 webserver.cc:460] Webserver started at http://127.1.166.1:42025/ using document root <none> and password file <none>
I20260620 22:57:19.578819 10453 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:19.578976 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:19.580447 10453 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:57:19.581064 10487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.581243 10453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:57:19.581332 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "4310d0b60ee14de39b19a19f9c842c60"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "8ac40979662b9e0fcdd0543c4d294c39"
server_key_iv: "a9a39ca66a36621134be3949660a5abc"
server_key_version: "encryptionkey@0"
I20260620 22:57:19.581725 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:19.605268 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:19.608139 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:19.608314 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:19.608605 10453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:19.609113 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:19.609229 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.609272 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:19.609288 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.616652 10453 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:42961
I20260620 22:57:19.616683 10600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:42961 every 8 connection(s)
I20260620 22:57:19.617079 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:57:19.622956 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 10453
I20260620 22:57:19.623255 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:42591
--local_ip_for_outbound_sockets=127.1.166.2
--tserver_master_addrs=127.1.166.62:38171
--webserver_port=43411
--webserver_interface=127.1.166.2
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.166.2
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:19.628319 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.617843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:40599)
0620 22:57:19.618033 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.618092 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.618827 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.619501 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.619505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.620268 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.620546 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.620552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.621853 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.621854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.623453 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.623456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.624351 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.624352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.624499 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.625137 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.625148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.627999 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
W20260620 22:57:19.628706 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.617975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:40599 (local address 127.1.166.62:38171)
0620 22:57:19.618600 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.618612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.618665 (+ 53us) server_negotiation.cc:407] Connection header received
0620 22:57:19.619154 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.619163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.619226 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.619400 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.620747 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.621734 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.623559 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.624228 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.624359 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.628516 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:40599: BlockingRecv error: recv got EOF from 127.1.166.1:40599 (error 108)
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":78,"threads_started":1}
W20260620 22:57:19.628875 10601 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:38171 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.166.62:38171: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:19.634048 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.629130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:55639)
0620 22:57:19.629169 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.629181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.629226 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.629364 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.629364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.629510 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.629563 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.629564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.630210 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.630210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.630840 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.630843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.631181 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.631182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.631251 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.631682 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.631688 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.633970 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 22:57:19.634204 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.629201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:55639 (local address 127.1.166.62:38171)
0620 22:57:19.629249 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.629251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.629261 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:57:19.629296 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.629297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.629309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.629353 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.629636 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.630132 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.630924 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.631118 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.631175 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.634129 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:55639: BlockingRecv error: recv got EOF from 127.1.166.1:55639 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:19.639832 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.634542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:48629)
0620 22:57:19.634633 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.634647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.634704 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.634838 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.634839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.634982 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.635027 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.635028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.635975 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.635977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.636498 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.636499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.637017 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.637019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.637093 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.637663 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.637668 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.639767 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 22:57:19.639959 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.634582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:48629 (local address 127.1.166.62:38171)
0620 22:57:19.634627 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.634629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.634687 (+ 58us) server_negotiation.cc:407] Connection header received
0620 22:57:19.634757 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.634758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.634770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.634832 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.635082 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.635874 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.636568 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.636873 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.636940 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.639891 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:48629: BlockingRecv error: recv got EOF from 127.1.166.1:48629 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 22:57:19.639989 10601 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:57:19.716485 10605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:19.716743 10605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:19.716773 10605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:19.718151 10605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:19.718211 10605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:19.718251 10605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:57:19.720048 10605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.2.keytab
--principal=oryx/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:42591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=43411
--tserver_master_addrs=127.1.166.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:19.720332 10605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:19.720549 10605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:19.726164 10610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.726182 10611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.726164 10613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:19.726202 10605 server_base.cc:1034] running on GCE node
I20260620 22:57:19.726612 10605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:19.726889 10605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:19.728129 10605 hybrid_clock.cc:648] HybridClock initialized: now 1781996239728088 us; error 33 us; skew 500 ppm
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996239, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:19.731709 10605 init.cc:377] Logged in from keytab as oryx/127.1.166.2@KRBTEST.COM (short username oryx)
I20260620 22:57:19.732288 10605 webserver.cc:460] Webserver started at http://127.1.166.2:43411/ using document root <none> and password file <none>
I20260620 22:57:19.732515 10605 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:19.732575 10605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:19.734308 10605 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:57:19.735234 10620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.735489 10605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:19.735610 10605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "37b0d415d78b4898a31b11e8b3cdf2f9"
format_stamp: "Formatted at 2026-06-20 22:57:18 on dist-test-slave-rdl9"
server_key: "bf98940f62ac0c125c66bfeb4d57d93c"
server_key_iv: "028afde5361cd1761770dbdfb81daf8c"
server_key_version: "encryptionkey@0"
I20260620 22:57:19.736073 10605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:19.747412 10605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:19.749879 10605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:19.750034 10605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:19.750284 10605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:19.750631 10605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:19.750672 10605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.750703 10605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:19.750728 10605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.757599 10605 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:42591
I20260620 22:57:19.757804 10733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:42591 every 8 connection(s)
I20260620 22:57:19.758044 10605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:57:19.759639 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 10605
I20260620 22:57:19.759965 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:36073
--local_ip_for_outbound_sockets=127.1.166.3
--tserver_master_addrs=127.1.166.62:38171
--webserver_port=38197
--webserver_interface=127.1.166.3
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.166.3
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:57:19.767624 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.758829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:37689 (local address 127.1.166.62:38171)
0620 22:57:19.758879 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.758881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.759157 (+ 276us) server_negotiation.cc:407] Connection header received
0620 22:57:19.759708 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.759710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.759731 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.759805 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.760875 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.761492 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.763040 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.763245 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.763322 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.767555 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:37689: BlockingRecv error: recv got EOF from 127.1.166.2:37689 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:57:19.767689 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.758831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:37689)
0620 22:57:19.758992 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.759031 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.759558 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.759909 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.759914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.760533 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.760751 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.760756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.761617 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.761619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.762916 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.762921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.763349 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.763350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.763509 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.764106 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.764118 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.767404 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
W20260620 22:57:19.767962 10734 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:38171 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.166.62:38171: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:19.773610 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.768215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:37025)
0620 22:57:19.768271 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.768282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.768323 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.768513 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.768515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.768662 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.768729 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.768731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.769407 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.769408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.769949 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.769951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.770273 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.770273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.770372 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.770822 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.770828 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.773517 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 22:57:19.773759 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.768321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:37025 (local address 127.1.166.62:38171)
0620 22:57:19.768398 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.768400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.768407 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:57:19.768441 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.768442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.768451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.768499 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.768799 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.769328 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.770022 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.770214 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.770271 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.773676 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:37025: BlockingRecv error: recv got EOF from 127.1.166.2:37025 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:19.779857 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.774210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:57881)
0620 22:57:19.774398 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.774411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.774458 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.774632 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.774633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.774798 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.774856 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.774858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.775628 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.775629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.776636 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.776643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.777065 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.777066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.777132 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.777681 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.777688 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.779796 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165}
W20260620 22:57:19.780078 10734 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:57:19.780138 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.774304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:57881 (local address 127.1.166.62:38171)
0620 22:57:19.774440 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.774442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.774460 (+ 18us) server_negotiation.cc:407] Connection header received
0620 22:57:19.774532 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.774533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.774543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.774612 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.775013 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.775546 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.776769 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.776981 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.777053 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.780065 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:57881: BlockingRecv error: recv got EOF from 127.1.166.2:57881 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260620 22:57:19.853484 10737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:19.853731 10737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:19.853756 10737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:19.855039 10737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:19.855088 10737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:19.855115 10737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:57:19.856612 10737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/krb5kdc/oryx_127.1.166.3.keytab
--principal=oryx/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:36073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=38197
--tserver_master_addrs=127.1.166.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:19.856812 10737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:19.857012 10737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:19.862094 10745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.862139 10743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:19.862082 10742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:19.862380 10737 server_base.cc:1034] running on GCE node
I20260620 22:57:19.862665 10737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:19.862927 10737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:19.864082 10737 hybrid_clock.cc:648] HybridClock initialized: now 1781996239864066 us; error 30 us; skew 500 ppm
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996239, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:19.866807 10737 init.cc:377] Logged in from keytab as oryx/127.1.166.3@KRBTEST.COM (short username oryx)
I20260620 22:57:19.867389 10737 webserver.cc:460] Webserver started at http://127.1.166.3:38197/ using document root <none> and password file <none>
I20260620 22:57:19.867651 10737 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:19.867712 10737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:19.869354 10737 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:57:19.870069 10752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.870252 10737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:19.870362 10737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "7d81376b7faf4d10a18f718113cbd00e"
format_stamp: "Formatted at 2026-06-20 22:57:19 on dist-test-slave-rdl9"
server_key: "9c89ebf286c07e65453b7ea4953f1c73"
server_key_iv: "23fe34a8e0532088768553c62cd08582"
server_key_version: "encryptionkey@0"
I20260620 22:57:19.870747 10737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:19.891732 10737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:19.894671 10737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:19.894815 10737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:19.895083 10737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:19.895401 10737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:19.895437 10737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.895466 10737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:19.895483 10737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:19.902065 10737 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:36073
I20260620 22:57:19.902114 10865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:36073 every 8 connection(s)
I20260620 22:57:19.902515 10737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:57:19.906512 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 10737
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:57:19.911762 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.903281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:46057 (local address 127.1.166.62:38171)
0620 22:57:19.903339 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.903341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.903660 (+ 319us) server_negotiation.cc:407] Connection header received
0620 22:57:19.904184 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.904185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.904197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.904262 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.905483 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.906098 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.907451 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.907743 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.907822 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.911707 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:46057: BlockingRecv error: recv got EOF from 127.1.166.3:46057 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:57:19.911871 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.903280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:46057)
0620 22:57:19.903495 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.903545 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.904048 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.904364 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.904368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.905078 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.905305 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.905310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.906240 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.906241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.907360 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.907365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.907852 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.907853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.907980 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.908525 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.908534 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.911633 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":101,"threads_started":1}
W20260620 22:57:19.912107 10866 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:38171 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.166.62:38171: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:19.916133 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.907107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.1.166.62:38171)
0620 22:57:19.907228 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.907230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.907236 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:57:19.907258 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.907259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.907270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.907326 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.908030 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.908654 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.909548 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.909830 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.909916 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.911366 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:19.911374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:19.911375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:19.911387 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:19.914257 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:19.914866 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:19.914869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:19.914870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:19.915002 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:19.915220 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:19.915221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:19.915222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:19.915568 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:19.915693 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:19.915914 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 22:57:19.916026 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:57:19.917915 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.912464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:43515 (local address 127.1.166.62:38171)
0620 22:57:19.912507 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.912509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.912536 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:57:19.912600 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.912600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.912610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.912652 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.912936 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.913475 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.914156 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.914452 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.914507 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.917868 (+ 3361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:43515: BlockingRecv error: recv got EOF from 127.1.166.3:43515 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:57:19.917857 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.912431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:43515)
0620 22:57:19.912481 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.912495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.912554 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.912665 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.912666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.912813 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.912866 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.912867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.913567 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.913568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.914075 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.914077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.914527 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.914528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.914596 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.915047 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.915052 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.917779 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:19 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:19.922940 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:19.918316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:33285)
0620 22:57:19.918404 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:19.918416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:19.918586 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:19.918723 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:19.918724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:19.918864 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:19.918907 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.918910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.919548 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.919549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.920042 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:19.920043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.920361 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.920361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:19.920410 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.920856 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:19.920861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:19.922854 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 22:57:19.923041 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:19.918405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:33285 (local address 127.1.166.62:38171)
0620 22:57:19.918564 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:19.918566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:19.918577 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:19.918651 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:19.918652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:19.918662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:19.918712 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:19.918978 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.919468 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.920116 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:19.920300 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:19.920357 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:19.922985 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:33285: BlockingRecv error: recv got EOF from 127.1.166.3:33285 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260620 22:57:19.923211 10866 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 22:57:20 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:20 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:20.645354 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:20.640545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:60661)
0620 22:57:20.640741 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:20.640755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:20.640840 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:20.641077 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:20.641078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:20.641258 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:20.641311 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:20.641313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.642017 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.642018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:20.642603 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:20.642605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.642989 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.642990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:20.643039 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:20.643563 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:20.643567 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:20.645269 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":129,"threads_started":1}
W20260620 22:57:20.645468 10871 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:20.640543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:60661 (local address 127.1.166.62:38171)
0620 22:57:20.640758 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:20.640760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:20.640906 (+ 146us) server_negotiation.cc:407] Connection header received
0620 22:57:20.640942 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:20.640943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:20.640956 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:20.641010 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:20.641412 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.641926 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.642700 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.642902 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.642964 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:20.645386 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:60661: BlockingRecv error: recv got EOF from 127.1.166.1:60661 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
Jun 20 22:57:20 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:20 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:20.785830 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:20.780638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:47127)
0620 22:57:20.780791 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:20.780803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:20.780867 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:20.781073 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:20.781074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:20.781254 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:20.781312 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:20.781314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.782024 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.782025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:20.782706 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:20.782709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.783162 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.783163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:20.783227 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:20.783741 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:20.783746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:20.785741 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
W20260620 22:57:20.786185 10871 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:20.780775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:47127 (local address 127.1.166.62:38171)
0620 22:57:20.780864 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:20.780866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:20.780924 (+ 58us) server_negotiation.cc:407] Connection header received
0620 22:57:20.780953 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:20.780953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:20.780966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:20.781013 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:20.781403 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.781920 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.782826 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.783061 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.783130 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:20.786109 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:47127: BlockingRecv error: recv got EOF from 127.1.166.2:47127 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 22:57:20 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:20 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:20.929281 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:20.923692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:55393)
0620 22:57:20.923830 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:20.923845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:20.924076 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:20.924241 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:20.924243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:20.924483 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:20.924553 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:20.924555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.925287 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.925289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:20.926046 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:20.926056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.926643 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.926645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:20.926712 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:20.927216 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:20.927221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:20.929182 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260620 22:57:20.929368 10871 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:20.923791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:55393 (local address 127.1.166.62:38171)
0620 22:57:20.923885 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:20.923887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:20.924027 (+ 140us) server_negotiation.cc:407] Connection header received
0620 22:57:20.924166 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:20.924168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:20.924186 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:20.924232 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:20.924622 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.925203 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.926247 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:20.926551 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:20.926638 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:20.929307 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:55393: BlockingRecv error: recv got EOF from 127.1.166.3:55393 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 22:57:21 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:21 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:21.650988 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:21.645931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:50203)
0620 22:57:21.646126 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:21.646138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:21.646206 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:21.646467 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:21.646468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:21.646630 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:21.646680 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:21.646682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.647405 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.647406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:21.647975 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:21.647977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.648419 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.648420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:21.648482 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:21.648987 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:21.648992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:21.650919 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
W20260620 22:57:21.651335 10876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:21.646156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:50203 (local address 127.1.166.62:38171)
0620 22:57:21.646277 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:21.646279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:21.646289 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:57:21.646316 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:21.646317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:21.646327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:21.646420 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:21.646786 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.647320 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.648056 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.648346 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.648409 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:21.651254 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:50203: BlockingRecv error: recv got EOF from 127.1.166.1:50203 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
Jun 20 22:57:21 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:21 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:21.791072 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:21.786392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:38645)
0620 22:57:21.786538 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:21.786548 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:21.786611 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:21.786801 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:21.786802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:21.786980 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:21.787032 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:21.787034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.787668 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.787669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:21.788194 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:21.788195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.788560 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.788561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:21.788599 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:21.789054 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:21.789058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:21.790998 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
W20260620 22:57:21.791203 10876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:21.786435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:38645 (local address 127.1.166.62:38171)
0620 22:57:21.786528 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:21.786529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:21.786607 (+ 78us) server_negotiation.cc:407] Connection header received
0620 22:57:21.786683 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:21.786684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:21.786696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:21.786752 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:21.787110 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.787596 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.788298 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.788494 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.788557 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:21.791138 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:38645: BlockingRecv error: recv got EOF from 127.1.166.2:38645 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 22:57:21 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:21 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:21.934952 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:21.930000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:35001)
0620 22:57:21.930236 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:21.930252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:21.930373 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:21.930527 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:21.930528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:21.930695 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:21.930745 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:21.930746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.931437 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.931438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:21.931984 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:21.931986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.932374 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.932376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:21.932425 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:21.932923 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:21.932928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:21.934874 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
W20260620 22:57:21.935151 10876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:21.930228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:35001 (local address 127.1.166.62:38171)
0620 22:57:21.930333 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:21.930359 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:21.930376 (+ 17us) server_negotiation.cc:407] Connection header received
0620 22:57:21.930427 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:21.930428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:21.930442 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:21.930504 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:21.930846 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.931356 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.932078 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:21.932280 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:21.932342 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:21.935058 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:35001: BlockingRecv error: recv got EOF from 127.1.166.3:35001 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 22:57:22 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:22 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:22.657070 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:22.651607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:51421)
0620 22:57:22.651768 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:22.651779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:22.651836 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:22.652022 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:22.652023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:22.652192 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:22.652238 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:22.652240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.652948 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.652949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:22.653505 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:22.653508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.654068 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.654069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:22.654133 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:22.654728 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:22.654733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:22.656967 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
W20260620 22:57:22.657193 10880 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:22.651740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:51421 (local address 127.1.166.62:38171)
0620 22:57:22.651849 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:22.651850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:22.651859 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:57:22.651897 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:22.651898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:22.651910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:22.651970 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:22.652342 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.652864 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.653660 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.653927 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.654032 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:22.657111 (+ 3079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:51421: BlockingRecv error: recv got EOF from 127.1.166.1:51421 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":45,"threads_started":1}
Jun 20 22:57:22 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:22 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:22.797125 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:22.791646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:33367)
0620 22:57:22.791806 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:22.791816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:22.791872 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:22.792059 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:22.792060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:22.792228 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:22.792275 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:22.792277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.793344 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.793346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:22.793951 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:22.793954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.794648 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.794650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:22.794729 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:22.795315 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:22.795320 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:22.797053 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
W20260620 22:57:22.797226 10880 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:22.791852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:33367 (local address 127.1.166.62:38171)
0620 22:57:22.791940 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:22.791942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:22.791951 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:57:22.791980 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:22.791981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:22.791993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:22.792039 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:22.792347 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.793177 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.794159 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.794512 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.794607 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:22.797155 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:33367: BlockingRecv error: recv got EOF from 127.1.166.2:33367 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 22:57:22 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:22 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:22.941901 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:22.935632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:44573)
0620 22:57:22.935847 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:22.935858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:22.935931 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:22.936234 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:22.936236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:22.936412 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:22.936560 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:22.936567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.937414 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.937416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:22.938005 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:22.938007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.938430 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.938431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:22.938485 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:22.939028 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:22.939035 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:22.941794 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
W20260620 22:57:22.942008 10880 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:22.935745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:44573 (local address 127.1.166.62:38171)
0620 22:57:22.935831 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:22.935846 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:22.936002 (+ 156us) server_negotiation.cc:407] Connection header received
0620 22:57:22.936056 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:22.936057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:22.936080 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:22.936167 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:22.936716 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.937276 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.938128 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:22.938331 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:22.938423 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:22.941940 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:44573: BlockingRecv error: recv got EOF from 127.1.166.3:44573 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 22:57:23 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:23 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:23.667111 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:23.657895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:45209)
0620 22:57:23.658064 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:23.658077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:23.658143 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:23.658530 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:23.658532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:23.658708 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:23.658769 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:23.658771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.659644 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.659645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:23.660235 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:23.660237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.660843 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.660844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:23.660900 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:23.664968 (+ 4068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:23.664975 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:23.667017 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
W20260620 22:57:23.667349 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:23.658055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:45209 (local address 127.1.166.62:38171)
0620 22:57:23.658231 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:23.658233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:23.658243 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:57:23.658276 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:23.658277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:23.658289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:23.658433 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:23.658904 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.659503 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.660405 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.660698 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.660778 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:23.667275 (+ 6497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:45209: BlockingRecv error: recv got EOF from 127.1.166.1:45209 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
Jun 20 22:57:23 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:23 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:23.803268 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:23.797821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:57231)
0620 22:57:23.798070 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:23.798085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:23.798162 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:23.798528 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:23.798529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:23.798752 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:23.798803 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:23.798805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.799515 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.799516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:23.800069 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:23.800071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.800418 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.800418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:23.800471 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:23.801022 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:23.801027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:23.803171 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
W20260620 22:57:23.803372 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:23.798010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:57231 (local address 127.1.166.62:38171)
0620 22:57:23.798164 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:23.798167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:23.798245 (+ 78us) server_negotiation.cc:407] Connection header received
0620 22:57:23.798293 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:23.798294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:23.798311 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:23.798516 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:23.798894 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.799434 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.800154 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.800347 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.800412 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:23.803315 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:57231: BlockingRecv error: recv got EOF from 127.1.166.2:57231 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 20 22:57:23 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:23 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:23.947575 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:23.942553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:57863)
0620 22:57:23.942746 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:23.942758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:23.942841 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:23.943019 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:23.943020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:23.943197 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:23.943247 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:23.943249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.944036 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.944038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:23.944860 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:23.944865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.945252 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.945253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:23.945303 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:23.945788 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:23.945793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:23.947513 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":125,"threads_started":1}
W20260620 22:57:23.947746 10884 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:23.942693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:57863 (local address 127.1.166.62:38171)
0620 22:57:23.942749 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:23.942751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:23.942864 (+ 113us) server_negotiation.cc:407] Connection header received
0620 22:57:23.942899 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:23.942900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:23.942914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:23.942972 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:23.943351 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.943897 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.944979 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:23.945179 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:23.945232 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:23.947673 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:57863: BlockingRecv error: recv got EOF from 127.1.166.3:57863 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 22:57:24 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:24 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:24.673017 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:24.667642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:38609)
0620 22:57:24.667770 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:24.667782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:24.667849 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:24.667985 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:24.667986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:24.668185 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:24.668243 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:24.668246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.668881 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.668882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:24.669419 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:24.669420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.669851 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.669852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:24.669907 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:24.670545 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:24.670550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:24.672767 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
W20260620 22:57:24.673064 10888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:24.667707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:38609 (local address 127.1.166.62:38171)
0620 22:57:24.667859 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:24.667861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:24.667872 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:24.667898 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:24.667898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:24.667908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:24.667964 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:24.668309 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.668806 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.669534 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.669754 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.669824 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:24.672927 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:38609: BlockingRecv error: recv got EOF from 127.1.166.1:38609 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
Jun 20 22:57:24 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:24 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:24.808665 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:24.803887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:40899)
0620 22:57:24.804076 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:24.804090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:24.804150 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:24.804265 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:24.804266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:24.804427 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:24.804473 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:24.804475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.805217 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.805219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:24.805853 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:24.805856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.806321 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.806322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:24.806447 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:24.806905 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:24.806909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:24.808602 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
W20260620 22:57:24.808840 10888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:24.804066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:40899 (local address 127.1.166.62:38171)
0620 22:57:24.804128 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:24.804130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:24.804137 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:57:24.804203 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:24.804203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:24.804213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:24.804255 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:24.804568 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.805073 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.805997 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.806209 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.806277 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:24.808756 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:40899: BlockingRecv error: recv got EOF from 127.1.166.2:40899 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 22:57:24 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:24 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:24.954144 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:24.948018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:39299)
0620 22:57:24.948164 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:24.948175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:24.948230 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:24.948377 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:24.948378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:24.948579 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:24.948641 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:24.948643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.949584 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.949585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:24.950381 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:24.950384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.951055 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.951056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:24.951153 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:24.951734 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:24.951740 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:24.954041 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260620 22:57:24.954282 10888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:24.948072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:39299 (local address 127.1.166.62:38171)
0620 22:57:24.948129 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:24.948131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:24.948219 (+ 88us) server_negotiation.cc:407] Connection header received
0620 22:57:24.948283 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:24.948284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:24.948300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:24.948365 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:24.948710 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.949486 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.950483 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:24.950847 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:24.951051 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:24.954202 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:39299: BlockingRecv error: recv got EOF from 127.1.166.3:39299 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 22:57:25 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:25 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:25.679601 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:25.673792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:58541)
0620 22:57:25.674224 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:25.674240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:25.674303 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:25.674825 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:25.674826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:25.675000 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:25.675054 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:25.675055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.675795 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.675796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:25.676603 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:25.676607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.677240 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.677242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:25.677310 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:25.677828 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:25.677833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:25.679521 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":290,"threads_started":1}
W20260620 22:57:25.679728 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:25.674046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:58541 (local address 127.1.166.62:38171)
0620 22:57:25.674576 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:25.674578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:25.674588 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:57:25.674615 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:25.674616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:25.674627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:25.674794 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:25.675167 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.675694 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.676721 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.677103 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.677183 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:25.679635 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:58541: BlockingRecv error: recv got EOF from 127.1.166.1:58541 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":126,"threads_started":1}
Jun 20 22:57:25 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:25 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:25.814560 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:25.809275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:60073)
0620 22:57:25.809408 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:25.809418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:25.809481 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:25.809616 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:25.809617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:25.809784 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:25.809832 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:25.809833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.810508 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.810509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:25.811036 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:25.811037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.811363 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.811364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:25.811416 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:25.811879 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:25.811884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:25.814389 (+ 2505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
W20260620 22:57:25.814666 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:25.809396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:60073 (local address 127.1.166.62:38171)
0620 22:57:25.809461 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:25.809463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:25.809474 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:25.809541 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:25.809541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:25.809558 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:25.809603 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:25.809903 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.810443 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.811113 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.811305 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.811358 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:25.814609 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:60073: BlockingRecv error: recv got EOF from 127.1.166.2:60073 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 22:57:25 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:25 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:25.960950 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:25.955616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:50581)
0620 22:57:25.955844 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:25.955867 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:25.955960 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:25.956128 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:25.956129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:25.956366 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:25.956421 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:25.956423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.957114 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.957115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:25.957688 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:25.957689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.958113 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.958115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:25.958200 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:25.958836 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:25.958842 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:25.960879 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260620 22:57:25.961122 10892 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:25.955755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:50581 (local address 127.1.166.62:38171)
0620 22:57:25.955818 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:25.955820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:25.955952 (+ 132us) server_negotiation.cc:407] Connection header received
0620 22:57:25.956014 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:25.956015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:25.956029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:25.956078 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:25.956526 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.957035 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.957788 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:25.957984 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:25.958042 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:25.961049 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:50581: BlockingRecv error: recv got EOF from 127.1.166.3:50581 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 22:57:26 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:26 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:26.685570 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:26.680228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:59713)
0620 22:57:26.680427 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:26.680439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:26.680512 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:26.680643 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:26.680644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:26.680810 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:26.680857 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:26.680859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.681842 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.681843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:26.682468 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:26.682471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.682874 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.682876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:26.682940 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:26.683413 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:26.683418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:26.685471 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
W20260620 22:57:26.685654 10896 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:26.680323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:59713 (local address 127.1.166.62:38171)
0620 22:57:26.680432 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:26.680434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:26.680504 (+ 70us) server_negotiation.cc:407] Connection header received
0620 22:57:26.680559 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:26.680560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:26.680571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:26.680628 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:26.681000 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.681713 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.682548 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.682761 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.682833 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:26.685591 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:59713: BlockingRecv error: recv got EOF from 127.1.166.1:59713 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
Jun 20 22:57:26 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:26 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:26.820714 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:26.815269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:44993)
0620 22:57:26.815444 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:26.815455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:26.815528 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:26.815658 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:26.815659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:26.815820 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:26.815881 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:26.815882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.816573 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.816574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:26.817134 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:26.817136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.817571 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.817573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:26.817657 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:26.818305 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:26.818310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:26.820625 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
W20260620 22:57:26.820789 10896 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:26.815391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:44993 (local address 127.1.166.62:38171)
0620 22:57:26.815443 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:26.815444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:26.815533 (+ 89us) server_negotiation.cc:407] Connection header received
0620 22:57:26.815581 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:26.815582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:26.815593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:26.815643 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:26.815953 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.816474 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.817218 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.817420 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.817475 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:26.820721 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:44993: BlockingRecv error: recv got EOF from 127.1.166.2:44993 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 22:57:26 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:26 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:26.966717 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:26.961657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:56471)
0620 22:57:26.961859 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:26.961874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:26.961971 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:26.962114 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:26.962115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:26.962306 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:26.962411 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:26.962414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.963118 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.963119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:26.963658 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:26.963660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.963989 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.963990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:26.964042 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:26.964530 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:26.964534 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:26.966635 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
W20260620 22:57:26.966794 10896 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:26.961814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:56471 (local address 127.1.166.62:38171)
0620 22:57:26.961900 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:26.961902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:26.961961 (+ 59us) server_negotiation.cc:407] Connection header received
0620 22:57:26.962032 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:26.962033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:26.962049 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:26.962103 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:26.962500 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.963035 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.963736 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:26.963925 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:26.963982 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:26.966737 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:56471: BlockingRecv error: recv got EOF from 127.1.166.3:56471 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 22:57:27 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:27 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:27.692483 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:27.686150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:57733)
0620 22:57:27.686309 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:27.686321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:27.686421 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:27.686655 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:27.686655 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:27.686821 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:27.686876 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:27.686878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.687639 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.687641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:27.688200 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:27.688202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.688704 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.688707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:27.688793 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:27.689463 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:27.689469 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:27.692397 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
W20260620 22:57:27.692598 10900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:27.686411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:57733 (local address 127.1.166.62:38171)
0620 22:57:27.686526 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:27.686528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:27.686536 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:27.686563 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:27.686564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:27.686573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:27.686640 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:27.686998 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.687526 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.688309 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.688521 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.688586 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:27.692514 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:57733: BlockingRecv error: recv got EOF from 127.1.166.1:57733 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
Jun 20 22:57:27 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:27 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:27.826941 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:27.821400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:42611)
0620 22:57:27.821570 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:27.821583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:27.821649 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:27.821796 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:27.821797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:27.821970 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:27.822020 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:27.822021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.822896 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.822897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:27.823445 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:27.823447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.823784 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.823785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:27.823838 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:27.824353 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:27.824358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:27.826867 (+ 2509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
W20260620 22:57:27.827113 10900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:27.821549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:42611 (local address 127.1.166.62:38171)
0620 22:57:27.821620 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:27.821622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:27.821634 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:57:27.821717 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:27.821718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:27.821733 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:27.821780 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:27.822091 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.822822 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.823527 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.823720 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.823777 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:27.827044 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:42611: BlockingRecv error: recv got EOF from 127.1.166.2:42611 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 22:57:27 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:27 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:27.973404 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:27.967315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:40323)
0620 22:57:27.967440 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:27.967453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:27.967515 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:27.967691 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:27.967692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:27.967860 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:27.967910 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:27.967912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.968693 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.968695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:27.969246 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:27.969249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.969836 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.969838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:27.969933 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:27.970650 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:27.970655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:27.973327 (+ 2672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
W20260620 22:57:27.974252 10900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:27.967429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:40323 (local address 127.1.166.62:38171)
0620 22:57:27.967487 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:27.967489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:27.967518 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:57:27.967577 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:27.967578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:27.967592 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:27.967643 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:27.968052 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.968560 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.969363 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:27.969675 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:27.969744 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:27.974162 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:40323: BlockingRecv error: recv got EOF from 127.1.166.3:40323 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 22:57:28 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:28 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:28.701249 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:28.693083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:55807)
0620 22:57:28.693211 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:28.693223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:28.693278 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:28.693694 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:28.693695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:28.693911 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:28.693978 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:28.693980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.695004 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.695005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:28.695551 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:28.695553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.695940 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.695940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:28.695980 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:28.699187 (+ 3207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:28.699192 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:28.701185 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
W20260620 22:57:28.701385 10904 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:28.693287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:55807 (local address 127.1.166.62:38171)
0620 22:57:28.693474 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:28.693476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:28.693488 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:57:28.693526 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:28.693527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:28.693542 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:28.693645 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:28.694130 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.694914 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.695661 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.695861 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.695944 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:28.701312 (+ 5368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:55807: BlockingRecv error: recv got EOF from 127.1.166.1:55807 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
Jun 20 22:57:28 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:28 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:28.832793 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:28.827538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:44279)
0620 22:57:28.827686 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:28.827697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:28.827781 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:28.827928 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:28.827929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:28.828107 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:28.828159 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:28.828160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.828847 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.828848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:28.829373 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:28.829375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.829734 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.829735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:28.829787 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:28.830290 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:28.830295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:28.832697 (+ 2402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
W20260620 22:57:28.832892 10904 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:28.827683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:44279 (local address 127.1.166.62:38171)
0620 22:57:28.827763 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:28.827765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:28.827777 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:57:28.827847 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:28.827849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:28.827861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:28.827915 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:28.828246 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.828755 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.829462 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.829666 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.829727 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:28.832831 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:44279: BlockingRecv error: recv got EOF from 127.1.166.2:44279 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 22:57:28 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:28 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:28.979280 10906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:28.974422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:44559)
0620 22:57:28.974610 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:28.974623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:28.974721 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:28.974896 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:28.974897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:28.975072 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:28.975122 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:28.975124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.975821 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.975822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:28.976379 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:28.976381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.976764 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.976765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:28.976811 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:28.977350 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:28.977354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:28.979215 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
W20260620 22:57:28.979414 10904 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:28.974550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:44559 (local address 127.1.166.62:38171)
0620 22:57:28.974643 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:28.974646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:28.974752 (+ 106us) server_negotiation.cc:407] Connection header received
0620 22:57:28.974794 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:28.974795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:28.974812 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:28.974883 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:28.975214 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.975723 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.976478 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:28.976684 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:28.976743 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:28.979326 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:44559: BlockingRecv error: recv got EOF from 127.1.166.3:44559 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 22:57:29 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:29 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:29.706727 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:29.701898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:52467)
0620 22:57:29.702085 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:29.702097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:29.702158 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:29.702298 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:29.702299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:29.702518 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:29.702570 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:29.702572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.703286 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.703287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:29.703851 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:29.703853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.704229 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.704230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:29.704284 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:29.704779 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:29.704784 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:29.706645 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
W20260620 22:57:29.706825 10908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:29.702031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:52467 (local address 127.1.166.62:38171)
0620 22:57:29.702136 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:29.702138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:29.702148 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:57:29.702214 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:29.702215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:29.702225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:29.702281 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:29.702672 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.703177 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.703952 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.704152 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.704207 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:29.706756 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:52467: BlockingRecv error: recv got EOF from 127.1.166.1:52467 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":46,"threads_started":1}
Jun 20 22:57:29 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:29 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:29.839247 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:29.833426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:47387)
0620 22:57:29.833589 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:29.833601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:29.833674 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:29.833804 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:29.833805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:29.833979 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:29.834027 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:29.834029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.834721 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.834722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:29.835267 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:29.835269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.835692 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.835693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:29.835747 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:29.836439 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:29.836447 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:29.839075 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
W20260620 22:57:29.839325 10908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:29.833558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:47387 (local address 127.1.166.62:38171)
0620 22:57:29.833619 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:29.833621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:29.833655 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:57:29.833732 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:29.833733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:29.833743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:29.833791 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:29.834087 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.834615 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.835379 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.835596 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.835661 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:29.839260 (+ 3599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:47387: BlockingRecv error: recv got EOF from 127.1.166.2:47387 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 22:57:29 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:29 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:29.984803 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:29.979772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:43655)
0620 22:57:29.979943 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:29.979954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:29.980004 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:29.980170 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:29.980170 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:29.980337 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:29.980382 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:29.980383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.981053 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.981054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:29.981599 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:29.981601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.982186 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.982187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:29.982245 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:29.982798 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:29.982803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:29.984706 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
W20260620 22:57:29.984965 10908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:29.979923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:43655 (local address 127.1.166.62:38171)
0620 22:57:29.979979 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:29.979981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:29.979992 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:29.980090 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:29.980091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:29.980105 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:29.980158 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:29.980464 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.980967 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.981752 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:29.982039 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:29.982144 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:29.984848 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:43655: BlockingRecv error: recv got EOF from 127.1.166.3:43655 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 22:57:30 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:30 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:30.713153 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:30.707357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:40415)
0620 22:57:30.707490 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:30.707499 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:30.707544 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:30.707731 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:30.707732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:30.707895 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:30.707940 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:30.707941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.708650 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.708652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:30.709179 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:30.709181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.709583 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.709584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:30.709629 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:30.710708 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:30.710714 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:30.713003 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
W20260620 22:57:30.713346 10912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:30.707495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:40415 (local address 127.1.166.62:38171)
0620 22:57:30.707607 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:30.707609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:30.707617 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:30.707641 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:30.707642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:30.707652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:30.707720 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:30.708028 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.708543 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.709295 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.709495 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.709551 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:30.713258 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:40415: BlockingRecv error: recv got EOF from 127.1.166.1:40415 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
Jun 20 22:57:30 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:30 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:30.844420 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:30.839782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:36089)
0620 22:57:30.839959 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:30.839971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:30.840034 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:30.840144 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:30.840145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:30.840313 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:30.840367 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:30.840368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.840998 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.840998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:30.841525 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:30.841526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.841862 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.841863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:30.841915 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:30.842420 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:30.842425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:30.844339 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
W20260620 22:57:30.844507 10912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:30.839851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:36089 (local address 127.1.166.62:38171)
0620 22:57:30.839912 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:30.839914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:30.840017 (+ 103us) server_negotiation.cc:407] Connection header received
0620 22:57:30.840089 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:30.840090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:30.840101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:30.840138 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:30.840417 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.840933 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.841588 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.841772 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.841820 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:30.844447 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:36089: BlockingRecv error: recv got EOF from 127.1.166.2:36089 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 22:57:30 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:30 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:30.990044 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:30.985379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:55371)
0620 22:57:30.985525 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:30.985538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:30.985606 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:30.985785 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:30.985786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:30.985949 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:30.985992 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:30.985993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.986775 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.986776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:30.987378 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:30.987380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.987775 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.987776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:30.987825 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:30.988277 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:30.988282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:30.989962 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
W20260620 22:57:30.990172 10912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:30.985484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:55371 (local address 127.1.166.62:38171)
0620 22:57:30.985545 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:30.985546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:30.985588 (+ 42us) server_negotiation.cc:407] Connection header received
0620 22:57:30.985666 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:30.985667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:30.985703 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:30.985746 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:30.986057 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.986665 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.987465 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:30.987676 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:30.987743 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:30.990106 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:55371: BlockingRecv error: recv got EOF from 127.1.166.3:55371 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 22:57:31 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:31 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:31.720623 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:31.713875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:57627)
0620 22:57:31.714034 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:31.714046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:31.714166 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:31.714377 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:31.714378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:31.714556 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:31.714673 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:31.714676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.716396 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.716398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:31.717113 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:31.717117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.717623 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.717625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:31.717715 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:31.718437 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:31.718443 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:31.720520 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
W20260620 22:57:31.720824 10916 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:31.714014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:57627 (local address 127.1.166.62:38171)
0620 22:57:31.714181 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:31.714183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:31.714210 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:57:31.714237 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:31.714237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:31.714249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:31.714302 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:31.714882 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.716199 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.717245 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.717466 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.717555 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:31.720730 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:57627: BlockingRecv error: recv got EOF from 127.1.166.1:57627 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
Jun 20 22:57:31 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:31 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:31.850000 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:31.844998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:48559)
0620 22:57:31.845164 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:31.845176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:31.845242 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:31.845415 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:31.845416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:31.845582 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:31.845630 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:31.845632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.846393 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.846394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:31.846935 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:31.846936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.847483 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.847484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:31.847550 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:31.848173 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:31.848178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:31.849933 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
W20260620 22:57:31.850193 10916 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:31.845160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:48559 (local address 127.1.166.62:38171)
0620 22:57:31.845229 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:31.845231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:31.845243 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:57:31.845297 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:31.845298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:31.845309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:31.845358 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:31.845749 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.846262 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.847090 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.847341 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.847443 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:31.850126 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:48559: BlockingRecv error: recv got EOF from 127.1.166.2:48559 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 22:57:31 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:31 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:31.996495 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:31.990775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:35191)
0620 22:57:31.990983 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:31.990995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:31.991074 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:31.991295 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:31.991296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:31.991506 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:31.991569 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:31.991571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.992427 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.992429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:31.993063 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:31.993067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.993564 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.993565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:31.993656 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:31.994258 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:31.994263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:31.996428 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
W20260620 22:57:31.996701 10916 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:31.990913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:35191 (local address 127.1.166.62:38171)
0620 22:57:31.990977 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:31.990979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:31.991127 (+ 148us) server_negotiation.cc:407] Connection header received
0620 22:57:31.991177 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:31.991178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:31.991201 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:31.991278 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:31.991708 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.992234 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.993197 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:31.993404 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:31.993473 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:31.996629 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:35191: BlockingRecv error: recv got EOF from 127.1.166.3:35191 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 22:57:32 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:32 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:32.726275 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:32.721349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:45155)
0620 22:57:32.721497 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:32.721507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:32.721562 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:32.721817 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:32.721819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:32.721989 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:32.722037 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:32.722039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.722893 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.722894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:32.723446 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:32.723448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.723820 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.723821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:32.723869 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:32.724357 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:32.724362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:32.726198 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
W20260620 22:57:32.726451 10920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:32.721479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:45155 (local address 127.1.166.62:38171)
0620 22:57:32.721595 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:32.721597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:32.721608 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:32.721641 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:32.721641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:32.721660 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:32.721760 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:32.722147 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.722769 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.723545 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.723745 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.723805 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:32.726377 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:45155: BlockingRecv error: recv got EOF from 127.1.166.1:45155 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
Jun 20 22:57:32 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:32 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:32.856745 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:32.850610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:49595)
0620 22:57:32.850775 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:32.850788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:32.850871 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:32.851005 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:32.851006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:32.851191 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:32.851234 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:32.851236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.851925 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.851926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:32.852536 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:32.852538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.853258 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.853259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:32.853318 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:32.853844 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:32.853852 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:32.856656 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
W20260620 22:57:32.856889 10920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:32.850761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:49595 (local address 127.1.166.62:38171)
0620 22:57:32.850819 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:32.850821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:32.850850 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:57:32.850934 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:32.850935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:32.850946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:32.850995 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:32.851302 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.851862 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.852887 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.853163 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.853250 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:32.856813 (+ 3563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:49595: BlockingRecv error: recv got EOF from 127.1.166.2:49595 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 22:57:32 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:32 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:33.002874 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:32.997246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:48029)
0620 22:57:32.997438 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:32.997451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:32.997555 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:32.997758 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:32.997760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:32.998023 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:32.998089 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:32.998092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.999096 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.999097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:32.999794 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:32.999797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.000276 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.000278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:33.000352 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:33.000839 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:33.000846 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:33.002797 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
W20260620 22:57:33.003149 10920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:32.997351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:48029 (local address 127.1.166.62:38171)
0620 22:57:32.997446 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:32.997450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:32.997546 (+ 96us) server_negotiation.cc:407] Connection header received
0620 22:57:32.997632 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:32.997633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:32.997653 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:32.997711 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:32.998208 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:32.998899 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:32.999911 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.000160 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.000230 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:33.003031 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:48029: BlockingRecv error: recv got EOF from 127.1.166.3:48029 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 22:57:33 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:33 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:33.735882 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:33.726823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:39275)
0620 22:57:33.726991 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:33.727003 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:33.727072 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:33.727268 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:33.727269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:33.727436 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:33.727486 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:33.727487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.728145 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.728146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:33.728702 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:33.728704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.729093 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.729094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:33.729147 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:33.733886 (+ 4739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:33.733894 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:33.735816 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
W20260620 22:57:33.736255 10924 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:33.726946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:39275 (local address 127.1.166.62:38171)
0620 22:57:33.727065 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:33.727066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:33.727077 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:33.727125 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:33.727126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:33.727136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:33.727196 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:33.727552 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.728058 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.728785 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.728993 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.729046 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:33.736167 (+ 7121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:39275: BlockingRecv error: recv got EOF from 127.1.166.1:39275 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
Jun 20 22:57:33 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:33 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:33.861869 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:33.857398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:36677)
0620 22:57:33.857532 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:33.857542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:33.857621 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:33.857774 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:33.857775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:33.857953 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:33.858000 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:33.858001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.858701 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.858702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:33.859219 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:33.859221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.859546 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.859546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:33.859598 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:33.860048 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:33.860052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:33.861763 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
W20260620 22:57:33.861903 10924 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:33.857561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:36677 (local address 127.1.166.62:38171)
0620 22:57:33.857652 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:33.857654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:33.857665 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:33.857697 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:33.857698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:33.857710 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:33.857759 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:33.858073 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.858625 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.859303 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:33.859487 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:33.859541 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:33.861850 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:36677: BlockingRecv error: recv got EOF from 127.1.166.2:36677 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 22:57:34 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:34 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.3@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:34.008607 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:34.003517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.3:44807)
0620 22:57:34.003692 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:34.003702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:34.003748 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:34.003877 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:34.003878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:34.004048 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:34.004095 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:34.004097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.004844 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.004846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:34.005395 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:34.005396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.005793 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.005794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:34.005838 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:34.006314 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:34.006319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:34.008511 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
W20260620 22:57:34.008669 10924 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:34.003660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:44807 (local address 127.1.166.62:38171)
0620 22:57:34.003726 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:34.003728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:34.003766 (+ 38us) server_negotiation.cc:407] Connection header received
0620 22:57:34.003802 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:34.003803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:34.003817 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:34.003862 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:34.004208 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.004742 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.005509 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.005708 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.005764 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:34.008617 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.3:44807: BlockingRecv error: recv got EOF from 127.1.166.3:44807 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 22:57:34 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:34 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.1@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:34.741401 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:34.736606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.1:51783)
0620 22:57:34.736756 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:34.736770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:34.736840 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:34.736996 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:34.736997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:34.737149 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:34.737199 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:34.737201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.737834 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.737835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:34.738431 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:34.738433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.738897 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.738898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:34.738962 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:34.739547 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:34.739551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:34.741311 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
W20260620 22:57:34.741500 10928 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:34.736743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:51783 (local address 127.1.166.62:38171)
0620 22:57:34.736879 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:34.736881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:34.736889 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:34.736914 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:34.736915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:34.736924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:34.736982 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:34.737253 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.737749 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.738545 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.738792 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.738870 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:34.741422 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.1:51783: BlockingRecv error: recv got EOF from 127.1.166.1:51783 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
Jun 20 22:57:34 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:57:34 dist-test-slave-rdl9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.166.2@KRBTEST.COM for oryx/127.1.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:57:34.867100 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:34.862487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:38171 (local address 127.1.166.2:53161)
0620 22:57:34.862644 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:34.862658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:34.862742 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:34.862899 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:34.862900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:34.863072 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:34.863122 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:34.863123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.863776 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.863776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:34.864297 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:34.864298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.864617 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.864617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:34.864668 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:34.865134 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:34.865139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:34.866994 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.166.62:38171: Server oryx/127.1.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
W20260620 22:57:34.867201 10928 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:34.862597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:53161 (local address 127.1.166.62:38171)
0620 22:57:34.862657 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:34.862660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:34.862726 (+ 66us) server_negotiation.cc:407] Connection header received
0620 22:57:34.862814 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:34.862815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:34.862829 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:34.862886 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:34.863199 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.863693 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.864372 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:34.864557 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:34.864612 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:34.867146 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.166.2:53161: BlockingRecv error: recv got EOF from 127.1.166.2:53161 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:57:34.915736 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 10453
I20260620 22:57:34.921211 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 10605
I20260620 22:57:34.926100 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 10737
I20260620 22:57:34.930560 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 10383
2026-06-20T22:57:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17637 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T22:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:57:34Z Disabled control of system clock
I20260620 22:57:34.960844 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35807
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:43201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:35807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 22:57:35.062819 10933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:35.063043 10933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:35.063063 10933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:35.064234 10933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:57:35.064276 10933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:35.064287 10933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:35.064299 10933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:57:35.064311 10933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:57:35.065651 10933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:35807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:35.065883 10933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:35.066143 10933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:35.070590 10941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.070604 10939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.070777 10933 server_base.cc:1034] running on GCE node
W20260620 22:57:35.070662 10938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.071094 10933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:35.071374 10933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:35.072540 10933 hybrid_clock.cc:648] HybridClock initialized: now 1781996255072528 us; error 32 us; skew 500 ppm
I20260620 22:57:35.073786 10933 webserver.cc:460] Webserver started at http://127.1.166.62:35045/ using document root <none> and password file <none>
I20260620 22:57:35.074014 10933 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:35.074061 10933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:35.074167 10933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:35.075297 10933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "d886215df50e40c290504aa7bd5811d6"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "fbfef2f679742a09785e9b8d258b1d8f"
server_key_iv: "6847f13b782343fcc89d25ea3e4af00b"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.075892 10933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "d886215df50e40c290504aa7bd5811d6"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "fbfef2f679742a09785e9b8d258b1d8f"
server_key_iv: "6847f13b782343fcc89d25ea3e4af00b"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.077931 10933 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 22:57:35.078828 10947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.079104 10933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:57:35.079190 10933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "d886215df50e40c290504aa7bd5811d6"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "fbfef2f679742a09785e9b8d258b1d8f"
server_key_iv: "6847f13b782343fcc89d25ea3e4af00b"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.079274 10933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:35.092299 10933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:35.092636 10933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:35.092765 10933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:35.098616 10933 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:35807
I20260620 22:57:35.098672 10999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:35807 every 8 connection(s)
I20260620 22:57:35.099053 10933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:57:35.099758 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:35.102440 11000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6: Bootstrap starting.
I20260620 22:57:35.103041 11000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:35.103318 11000 log.cc:824] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:35.104004 11000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6: No bootstrap required, opened a new log
I20260620 22:57:35.105574 11000 raft_consensus.cc:348] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d886215df50e40c290504aa7bd5811d6" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35807 } }
I20260620 22:57:35.105737 11000 raft_consensus.cc:374] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:35.105762 11000 raft_consensus.cc:729] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886215df50e40c290504aa7bd5811d6, State: Initialized, Role: FOLLOWER
I20260620 22:57:35.105906 11000 consensus_queue.cc:260] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d886215df50e40c290504aa7bd5811d6" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35807 } }
I20260620 22:57:35.105981 11000 raft_consensus.cc:388] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:57:35.106009 11000 raft_consensus.cc:482] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:57:35.106282 11000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:35.107241 11000 raft_consensus.cc:504] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d886215df50e40c290504aa7bd5811d6" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35807 } }
I20260620 22:57:35.107372 11000 leader_election.cc:302] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d886215df50e40c290504aa7bd5811d6; no voters:
I20260620 22:57:35.107555 11000 leader_election.cc:288] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:57:35.107659 11003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:35.107831 11000 sys_catalog.cc:564] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:57:35.107825 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 10933
I20260620 22:57:35.107878 11003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [term 1 LEADER]: Becoming Leader. State: Replica: d886215df50e40c290504aa7bd5811d6, State: Running, Role: LEADER
I20260620 22:57:35.107909 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:57:35.108031 11003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d886215df50e40c290504aa7bd5811d6" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35807 } }
I20260620 22:57:35.108115 1688 external_mini_cluster.cc:1428] Setting key d1d4d8dc535e00235274b1a70fa137a5
I20260620 22:57:35.108560 11004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d886215df50e40c290504aa7bd5811d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d886215df50e40c290504aa7bd5811d6" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35807 } } }
I20260620 22:57:35.108649 11004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:35.109061 11013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:57:35.109300 11005 sys_catalog.cc:455] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d886215df50e40c290504aa7bd5811d6. Latest consensus state: current_term: 1 leader_uuid: "d886215df50e40c290504aa7bd5811d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d886215df50e40c290504aa7bd5811d6" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35807 } } }
I20260620 22:57:35.109424 11005 sys_catalog.cc:458] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:35.109596 11013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:57:35.111173 11013 catalog_manager.cc:1347] Generated new cluster ID: 0a90a101569941dda7414319533ba90f
I20260620 22:57:35.111229 11013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:57:35.114255 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.108796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.1.166.62:35807)
0620 22:57:35.109723 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:35.109733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:35.109763 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:57:35.109867 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:35.109873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:35.109928 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:35.110050 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:35.110257 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.111494 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.112290 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.112753 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.112880 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:35.113570 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:35.113574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:35.113575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:57:35.113773 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:35.113813 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:57:35.113825 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:35.113884 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:35.113963 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 22:57:35.114065 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":63,"threads_started":1}
I20260620 22:57:35.129819 11013 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:57:35.130366 11013 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:57:35.136494 11013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d886215df50e40c290504aa7bd5811d6: Generated new TSK 0
I20260620 22:57:35.136744 11013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:57:35.142640 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35807
--builtin_ntp_servers=127.1.166.20:43201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 22:57:35.238240 11024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:35.238548 11024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:35.238574 11024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:35.239923 11024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:35.239989 11024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:35.240020 11024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:57:35.241868 11024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:35.242161 11024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:35.242530 11024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:35.248848 11030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.249032 11032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.248857 11029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.249500 11024 server_base.cc:1034] running on GCE node
I20260620 22:57:35.249730 11024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:35.250020 11024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:35.251215 11024 hybrid_clock.cc:648] HybridClock initialized: now 1781996255251205 us; error 41 us; skew 500 ppm
I20260620 22:57:35.252579 11024 webserver.cc:460] Webserver started at http://127.1.166.1:39041/ using document root <none> and password file <none>
I20260620 22:57:35.252835 11024 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:35.252890 11024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:35.253031 11024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:35.254204 11024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "fd116898626946adb25222cfcd054ec3"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "73af29c2c500649102ff41e2c52c2eae"
server_key_iv: "d1f22fef0f40bcfddfc83a507a250d0e"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.254627 11024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "fd116898626946adb25222cfcd054ec3"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "73af29c2c500649102ff41e2c52c2eae"
server_key_iv: "d1f22fef0f40bcfddfc83a507a250d0e"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.256798 11024 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:57:35.258188 11038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.258522 11024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:35.258661 11024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "fd116898626946adb25222cfcd054ec3"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "73af29c2c500649102ff41e2c52c2eae"
server_key_iv: "d1f22fef0f40bcfddfc83a507a250d0e"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.258761 11024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:35.271661 11024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:35.271993 11024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:35.272125 11024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:35.272361 11024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:35.272899 11024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:35.272953 11024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.272992 11024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:35.273008 11024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.278780 11024 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:41367
I20260620 22:57:35.278818 11151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:41367 every 8 connection(s)
I20260620 22:57:35.279142 11024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:57:35.284418 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.279907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:36117 (local address 127.1.166.62:35807)
0620 22:57:35.280003 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:35.280006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:35.280201 (+ 195us) server_negotiation.cc:407] Connection header received
0620 22:57:35.280714 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:35.280715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:35.280726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:35.280771 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:35.281419 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.281940 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.283038 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.283232 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.283289 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:35.284139 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:35.284140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:35.284140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:57:35.284145 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:35.284159 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:57:35.284162 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:35.284207 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:35.284327 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 22:57:35.284353 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 22:57:35.284616 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.279788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35807 (local address 127.1.166.1:36117)
0620 22:57:35.280083 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:35.280121 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:35.280633 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:35.280831 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:35.280834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:35.280842 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:35.281321 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:35.281325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.282053 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.282054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:35.282953 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:35.282956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.283334 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.283335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:35.283457 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:35.284010 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:57:35.284010 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:35.284039 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:57:35.284064 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:35.284237 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:35.284243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:35.284245 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:57:35.284324 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 22:57:35.284411 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260620 22:57:35.285233 11152 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35807
I20260620 22:57:35.285326 11152 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:35.285485 11152 heartbeater.cc:511] Master 127.1.166.62:35807 requested a full tablet report, sending...
I20260620 22:57:35.286054 10964 ts_manager.cc:194] Registered new tserver with Master: fd116898626946adb25222cfcd054ec3 (127.1.166.1:41367)
I20260620 22:57:35.286764 10964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.1:36117
I20260620 22:57:35.288725 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 11024
I20260620 22:57:35.288818 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:57:35.288998 1688 external_mini_cluster.cc:1428] Setting key 598503e8ef2a4ebb28d56bc8ef060484
I20260620 22:57:35.289812 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35807
--builtin_ntp_servers=127.1.166.20:43201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 22:57:35.378180 11155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:35.378489 11155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:35.378518 11155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:35.379781 11155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:35.379827 11155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:35.379854 11155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:57:35.382396 11155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:35.382690 11155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:35.383001 11155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:35.387982 11160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.387969 11161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.388068 11155 server_base.cc:1034] running on GCE node
W20260620 22:57:35.387969 11163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.388394 11155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:35.388659 11155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:35.389869 11155 hybrid_clock.cc:648] HybridClock initialized: now 1781996255389838 us; error 46 us; skew 500 ppm
I20260620 22:57:35.391222 11155 webserver.cc:460] Webserver started at http://127.1.166.2:39233/ using document root <none> and password file <none>
I20260620 22:57:35.391440 11155 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:35.391486 11155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:35.391573 11155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:35.392642 11155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "80e1da1836be4b36b6d3fbd75976abb6"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "30b504488789787cd4b8d8309feb1569"
server_key_iv: "5af0cc4c240e89d75c266f139bf57b92"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.392939 11155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "80e1da1836be4b36b6d3fbd75976abb6"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "30b504488789787cd4b8d8309feb1569"
server_key_iv: "5af0cc4c240e89d75c266f139bf57b92"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.394297 11155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:57:35.395488 11169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.395718 11155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:35.395815 11155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "80e1da1836be4b36b6d3fbd75976abb6"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "30b504488789787cd4b8d8309feb1569"
server_key_iv: "5af0cc4c240e89d75c266f139bf57b92"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.395890 11155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:35.411631 11155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:35.412241 11155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:35.412462 11155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:35.413326 11155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:35.413717 11155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:35.413754 11155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.413779 11155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:35.413790 11155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.419878 11155 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:43467
I20260620 22:57:35.419924 11282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:43467 every 8 connection(s)
I20260620 22:57:35.420238 11155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:57:35.425247 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 11155
I20260620 22:57:35.425338 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:57:35.425537 1688 external_mini_cluster.cc:1428] Setting key 1a9f2e62ada35256fe92f21ab5c13f43
I20260620 22:57:35.426548 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.421326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:47461 (local address 127.1.166.62:35807)
0620 22:57:35.421415 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:35.421417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:35.421675 (+ 258us) server_negotiation.cc:407] Connection header received
0620 22:57:35.422161 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:35.422162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:35.422173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:35.422229 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:35.422977 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.423476 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.424562 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.424782 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.424848 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:35.426113 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:35.426114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:35.426115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:57:35.426120 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:35.426141 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:57:35.426145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:35.426213 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:35.426430 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 22:57:35.426465 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 22:57:35.426687 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35807
--builtin_ntp_servers=127.1.166.20:43201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 22:57:35.426805 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.421314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35807 (local address 127.1.166.2:47461)
0620 22:57:35.421517 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:35.421551 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:35.422045 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:35.422294 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:35.422297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:35.422307 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:35.422870 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:35.422876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.423587 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.423588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:35.424443 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:35.424446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.424945 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.424947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:35.425184 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:35.425864 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:57:35.425865 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:35.425924 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:57:35.425970 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:35.426262 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:35.426271 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:35.426276 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:57:35.426421 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 22:57:35.426556 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260620 22:57:35.427328 11283 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35807
I20260620 22:57:35.427428 11283 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:35.427613 11283 heartbeater.cc:511] Master 127.1.166.62:35807 requested a full tablet report, sending...
I20260620 22:57:35.427966 10964 ts_manager.cc:194] Registered new tserver with Master: 80e1da1836be4b36b6d3fbd75976abb6 (127.1.166.2:43467)
I20260620 22:57:35.428396 10964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.2:47461
W20260620 22:57:35.513612 11286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:35.513819 11286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:35.513841 11286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:35.515028 11286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:35.515077 11286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:35.515103 11286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:57:35.516479 11286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:35.516799 11286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:35.517118 11286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:35.521888 11292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.522006 11286 server_base.cc:1034] running on GCE node
W20260620 22:57:35.521881 11294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.521891 11291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.522329 11286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:35.522634 11286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:35.523795 11286 hybrid_clock.cc:648] HybridClock initialized: now 1781996255523772 us; error 34 us; skew 500 ppm
I20260620 22:57:35.525082 11286 webserver.cc:460] Webserver started at http://127.1.166.3:35405/ using document root <none> and password file <none>
I20260620 22:57:35.525326 11286 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:35.525375 11286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:35.525501 11286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:35.526619 11286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "35440b0ba4564cf087e3a61031e578f5"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "d029230b0906cbfdc7a01d2d89d77ad3"
server_key_iv: "dd5febd981bd53795792bc44d71f1cb4"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.526958 11286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "35440b0ba4564cf087e3a61031e578f5"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "d029230b0906cbfdc7a01d2d89d77ad3"
server_key_iv: "dd5febd981bd53795792bc44d71f1cb4"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.528285 11286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:35.529217 11300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.529414 11286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:35.529492 11286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "35440b0ba4564cf087e3a61031e578f5"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "d029230b0906cbfdc7a01d2d89d77ad3"
server_key_iv: "dd5febd981bd53795792bc44d71f1cb4"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.529556 11286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:35.544425 11286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:35.544842 11286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:35.545147 11286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:35.545496 11286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:35.545861 11286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:35.545897 11286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.545926 11286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:35.545951 11286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.551502 11286 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:41115
I20260620 22:57:35.551585 11413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:41115 every 8 connection(s)
I20260620 22:57:35.551927 11286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:57:35.552832 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 11286
I20260620 22:57:35.552938 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:57:35.553385 1688 external_mini_cluster.cc:1428] Setting key fa030921232ce1d7ed8a3707a3fd50f9
I20260620 22:57:35.558604 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.552932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:40195 (local address 127.1.166.62:35807)
0620 22:57:35.552987 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:35.552989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:35.553410 (+ 421us) server_negotiation.cc:407] Connection header received
0620 22:57:35.553962 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:35.553963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:35.553974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:35.554021 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:35.555176 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.555693 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.556986 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.557186 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.557244 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:35.558311 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:35.558313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:35.558313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:57:35.558318 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:35.558358 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:57:35.558365 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:35.558426 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:35.558514 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:57:35.558541 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:57:35.558871 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.552817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35807 (local address 127.1.166.3:40195)
0620 22:57:35.553234 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:35.553287 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:35.553870 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:35.554201 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:35.554207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:35.554218 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:35.555061 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:35.555071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.555821 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.555823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:35.556856 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:35.556860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:35.557298 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:35.557299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:35.557451 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:35.558108 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:57:35.558109 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:35.558147 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:57:35.558178 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:35.558441 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:35.558444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:35.558448 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:57:35.558508 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:57:35.558619 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
I20260620 22:57:35.559587 11414 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35807
I20260620 22:57:35.559823 11414 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:35.560088 11414 heartbeater.cc:511] Master 127.1.166.62:35807 requested a full tablet report, sending...
I20260620 22:57:35.560557 10964 ts_manager.cc:194] Registered new tserver with Master: 35440b0ba4564cf087e3a61031e578f5 (127.1.166.3:41115)
I20260620 22:57:35.560914 10964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.3:40195
I20260620 22:57:35.566318 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:57:35.568393 11015 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:35.567789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.1.166.62:35807)
0620 22:57:35.567876 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:35.567878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:35.567963 (+ 85us) server_negotiation.cc:407] Connection header received
0620 22:57:35.568076 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:35.568077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:35.568093 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:35.568143 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:35.568320 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55768: BlockingRecv error: recv got EOF from 127.0.0.1:55768 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 22:57:35.568553 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 11024
I20260620 22:57:35.573269 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 11155
I20260620 22:57:35.578905 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 11286
I20260620 22:57:35.583606 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 10933
2026-06-20T22:57:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (651 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T22:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:57:35Z Disabled control of system clock
I20260620 22:57:35.609859 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:43193
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:43193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:57:35.692982 11425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:35.693202 11425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:35.693223 11425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:35.694380 11425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:57:35.694417 11425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:35.694428 11425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:35.694439 11425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:57:35.694450 11425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:57:35.695777 11425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:43193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:43193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:35.696000 11425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:35.696245 11425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:35.700912 11430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.700938 11433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.700922 11431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.701093 11425 server_base.cc:1034] running on GCE node
I20260620 22:57:35.701325 11425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:35.701618 11425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:35.702790 11425 hybrid_clock.cc:648] HybridClock initialized: now 1781996255702774 us; error 29 us; skew 500 ppm
I20260620 22:57:35.704047 11425 webserver.cc:460] Webserver started at http://127.1.166.62:39845/ using document root <none> and password file <none>
I20260620 22:57:35.704264 11425 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:35.704313 11425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:35.704437 11425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:35.705546 11425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "8838e8c823cf4602b0885106115775fb"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "4d8eff3ef35c3cf124c2b32fb7e928ac"
server_key_iv: "9fca90a945b6d4965716084809d818cb"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.705888 11425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "8838e8c823cf4602b0885106115775fb"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "4d8eff3ef35c3cf124c2b32fb7e928ac"
server_key_iv: "9fca90a945b6d4965716084809d818cb"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.707247 11425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:57:35.708196 11439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.708467 11425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:35.708570 11425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "8838e8c823cf4602b0885106115775fb"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "4d8eff3ef35c3cf124c2b32fb7e928ac"
server_key_iv: "9fca90a945b6d4965716084809d818cb"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.708643 11425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:35.712237 11425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:35.712519 11425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:35.712641 11425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:35.716650 11491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:43193 every 8 connection(s)
I20260620 22:57:35.716718 11425 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:43193
I20260620 22:57:35.717370 11425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:57:35.718075 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:35.720780 11492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb: Bootstrap starting.
I20260620 22:57:35.721346 11492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:35.721624 11492 log.cc:824] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:35.722263 11492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb: No bootstrap required, opened a new log
I20260620 22:57:35.723860 11492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8838e8c823cf4602b0885106115775fb" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43193 } }
I20260620 22:57:35.724005 11492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:35.724027 11492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8838e8c823cf4602b0885106115775fb, State: Initialized, Role: FOLLOWER
I20260620 22:57:35.724241 11492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8838e8c823cf4602b0885106115775fb" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43193 } }
I20260620 22:57:35.724344 11492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:57:35.724370 11492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:57:35.724411 11492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:35.724586 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 11425
I20260620 22:57:35.724663 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:57:35.724838 1688 external_mini_cluster.cc:1428] Setting key 67a4d514d97616db0ee899059dc30286
I20260620 22:57:35.725216 11492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8838e8c823cf4602b0885106115775fb" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43193 } }
I20260620 22:57:35.725406 11492 leader_election.cc:302] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8838e8c823cf4602b0885106115775fb; no voters:
I20260620 22:57:35.725590 11492 leader_election.cc:288] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:57:35.725612 11497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:35.725737 11497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [term 1 LEADER]: Becoming Leader. State: Replica: 8838e8c823cf4602b0885106115775fb, State: Running, Role: LEADER
I20260620 22:57:35.725864 11492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:57:35.725831 11497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8838e8c823cf4602b0885106115775fb" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43193 } }
I20260620 22:57:35.726572 11498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8838e8c823cf4602b0885106115775fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8838e8c823cf4602b0885106115775fb" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43193 } } }
I20260620 22:57:35.726675 11498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:35.726807 11499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8838e8c823cf4602b0885106115775fb. Latest consensus state: current_term: 1 leader_uuid: "8838e8c823cf4602b0885106115775fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8838e8c823cf4602b0885106115775fb" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 43193 } } }
I20260620 22:57:35.726866 11499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:35.726974 11506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:57:35.727382 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.725321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.1.166.62:43193)
0620 22:57:35.725565 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:35.725573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:35.725599 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:57:35.725717 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:35.725725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:35.725813 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:35.726206 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:35.726779 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:35.726782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:35.726783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:57:35.726786 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:35.726826 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:57:35.726839 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:35.727108 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:35.727140 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 22:57:35.727198 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260620 22:57:35.727689 11506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:57:35.729609 11506 catalog_manager.cc:1347] Generated new cluster ID: 8cf542745ec14951adada93a0f6315f7
I20260620 22:57:35.729666 11506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:57:35.752375 11506 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:57:35.752573 11506 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:57:35.757148 11506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8838e8c823cf4602b0885106115775fb: Generated new TSK 0
I20260620 22:57:35.757370 11506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:57:35.760006 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43193
--builtin_ntp_servers=127.1.166.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:57:35.848846 11516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:35.849074 11516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:35.849097 11516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:35.850373 11516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:35.850436 11516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:35.850466 11516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:57:35.852102 11516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:35.852398 11516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:35.852728 11516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:35.857956 11521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.857962 11522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.858204 11516 server_base.cc:1034] running on GCE node
W20260620 22:57:35.857960 11524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.858533 11516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:35.858817 11516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:35.860051 11516 hybrid_clock.cc:648] HybridClock initialized: now 1781996255860010 us; error 87 us; skew 500 ppm
I20260620 22:57:35.861488 11516 webserver.cc:460] Webserver started at http://127.1.166.1:46067/ using document root <none> and password file <none>
I20260620 22:57:35.861758 11516 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:35.861807 11516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:35.861953 11516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:35.863112 11516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "90eff96d88f64788be0867678139ece0"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "cb2dcbd674159855504d4f0e2cb2871d"
server_key_iv: "b3a0ef2aca7d4ae0a0aa7c61a457732d"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.863462 11516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "90eff96d88f64788be0867678139ece0"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "cb2dcbd674159855504d4f0e2cb2871d"
server_key_iv: "b3a0ef2aca7d4ae0a0aa7c61a457732d"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.864743 11516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:57:35.865556 11530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.865775 11516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:35.865875 11516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "90eff96d88f64788be0867678139ece0"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "cb2dcbd674159855504d4f0e2cb2871d"
server_key_iv: "b3a0ef2aca7d4ae0a0aa7c61a457732d"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.865962 11516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:35.869441 11516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:35.869741 11516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:35.869869 11516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:35.870078 11516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:35.870433 11516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:35.870471 11516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.870499 11516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:35.870523 11516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.876682 11516 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:37799
I20260620 22:57:35.876751 11643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:37799 every 8 connection(s)
I20260620 22:57:35.877054 11516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:57:35.880386 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.877788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:49051 (local address 127.1.166.62:43193)
0620 22:57:35.877874 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:35.877877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:35.878048 (+ 171us) server_negotiation.cc:407] Connection header received
0620 22:57:35.878598 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:35.878600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:35.878608 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:35.878660 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:35.880061 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:35.880063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:35.880063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:57:35.880068 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:35.880087 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:57:35.880092 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:35.880165 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:35.880286 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 22:57:35.880303 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 22:57:35.880597 11646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.877720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:43193 (local address 127.1.166.1:49051)
0620 22:57:35.877921 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:35.877957 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:35.878501 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:35.878734 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:35.878738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:35.878747 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:35.879852 (+ 1105us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:57:35.879853 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:35.879910 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:57:35.879942 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:35.880190 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:35.880197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:35.880200 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:57:35.880291 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 22:57:35.880361 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260620 22:57:35.881383 11644 heartbeater.cc:347] Connected to a master server at 127.1.166.62:43193
I20260620 22:57:35.881551 11644 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:35.881773 11644 heartbeater.cc:511] Master 127.1.166.62:43193 requested a full tablet report, sending...
I20260620 22:57:35.882521 11456 ts_manager.cc:194] Registered new tserver with Master: 90eff96d88f64788be0867678139ece0 (127.1.166.1:37799)
I20260620 22:57:35.885247 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 11516
I20260620 22:57:35.885325 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:57:35.885522 1688 external_mini_cluster.cc:1428] Setting key e107e1fc5e3fb27f7a6765240698ad37
I20260620 22:57:35.886606 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43193
--builtin_ntp_servers=127.1.166.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:57:35.969429 11647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:35.969683 11647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:35.969722 11647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:35.971189 11647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:35.971256 11647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:35.971299 11647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:57:35.972910 11647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:35.973220 11647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:35.973529 11647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:35.978435 11655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.978443 11653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:35.978555 11652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:35.979005 11647 server_base.cc:1034] running on GCE node
I20260620 22:57:35.979221 11647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:35.979460 11647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:35.980633 11647 hybrid_clock.cc:648] HybridClock initialized: now 1781996255980595 us; error 57 us; skew 500 ppm
I20260620 22:57:35.982182 11647 webserver.cc:460] Webserver started at http://127.1.166.2:42361/ using document root <none> and password file <none>
I20260620 22:57:35.982532 11647 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:35.982594 11647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:35.982736 11647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:35.984005 11647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "5cbeaf6f3f044120960d361d763e4576"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "b1913f15c6c7a0db41082751017481b4"
server_key_iv: "f3555c251c2d93d741c3b5e7a9a43b7e"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.984388 11647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "5cbeaf6f3f044120960d361d763e4576"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "b1913f15c6c7a0db41082751017481b4"
server_key_iv: "f3555c251c2d93d741c3b5e7a9a43b7e"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.985996 11647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:57:35.987088 11661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.987313 11647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:35.987416 11647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "5cbeaf6f3f044120960d361d763e4576"
format_stamp: "Formatted at 2026-06-20 22:57:35 on dist-test-slave-rdl9"
server_key: "b1913f15c6c7a0db41082751017481b4"
server_key_iv: "f3555c251c2d93d741c3b5e7a9a43b7e"
server_key_version: "encryptionkey@0"
I20260620 22:57:35.987499 11647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:35.991145 11647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:35.991438 11647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:35.991565 11647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:35.991768 11647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:35.992074 11647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:35.992105 11647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.992133 11647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:35.992157 11647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:35.997959 11647 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:45877
I20260620 22:57:35.998092 11774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:45877 every 8 connection(s)
I20260620 22:57:35.998420 11647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:57:36.001446 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 11647
I20260620 22:57:36.001557 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:57:36.001775 1688 external_mini_cluster.cc:1428] Setting key 9bbb153feced8af16b220d7b2b5eab9e
I20260620 22:57:36.001791 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.999189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:37903 (local address 127.1.166.62:43193)
0620 22:57:35.999267 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:35.999269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:35.999504 (+ 235us) server_negotiation.cc:407] Connection header received
0620 22:57:36.000134 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:36.000136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:36.000146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:36.000243 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:36.001490 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:36.001493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:36.001493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:57:36.001501 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:36.001530 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:57:36.001536 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:36.001615 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:36.001703 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:57:36.001721 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:57:36.001994 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:35.999187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:43193 (local address 127.1.166.2:37903)
0620 22:57:35.999380 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:35.999422 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:35.999927 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:36.000323 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:36.000327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:36.000336 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:36.001279 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:57:36.001280 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:36.001312 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:57:36.001344 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:36.001617 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:36.001624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:36.001627 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:57:36.001709 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:57:36.001770 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260620 22:57:36.002437 11775 heartbeater.cc:347] Connected to a master server at 127.1.166.62:43193
I20260620 22:57:36.002545 11775 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:36.002698 11775 heartbeater.cc:511] Master 127.1.166.62:43193 requested a full tablet report, sending...
I20260620 22:57:36.002694 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43193
--builtin_ntp_servers=127.1.166.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 22:57:36.003085 11455 ts_manager.cc:194] Registered new tserver with Master: 5cbeaf6f3f044120960d361d763e4576 (127.1.166.2:45877)
W20260620 22:57:36.088660 11778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:36.088917 11778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:36.088953 11778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:36.090278 11778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:36.090332 11778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:36.090399 11778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:57:36.091957 11778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:43193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:36.092216 11778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:36.092784 11778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:36.099423 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:36.099411 11784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:36.099411 11783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:36.099854 11778 server_base.cc:1034] running on GCE node
I20260620 22:57:36.100078 11778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:36.100370 11778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:36.101580 11778 hybrid_clock.cc:648] HybridClock initialized: now 1781996256101564 us; error 32 us; skew 500 ppm
I20260620 22:57:36.103143 11778 webserver.cc:460] Webserver started at http://127.1.166.3:43759/ using document root <none> and password file <none>
I20260620 22:57:36.103381 11778 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:36.103433 11778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:36.103559 11778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:36.104648 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "6b9a0224831e43b598499664f08de4cb"
format_stamp: "Formatted at 2026-06-20 22:57:36 on dist-test-slave-rdl9"
server_key: "14c7ecfe7829db1a61675d1c68863542"
server_key_iv: "654da15793f17e3460abe17805ca8708"
server_key_version: "encryptionkey@0"
I20260620 22:57:36.104983 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "6b9a0224831e43b598499664f08de4cb"
format_stamp: "Formatted at 2026-06-20 22:57:36 on dist-test-slave-rdl9"
server_key: "14c7ecfe7829db1a61675d1c68863542"
server_key_iv: "654da15793f17e3460abe17805ca8708"
server_key_version: "encryptionkey@0"
I20260620 22:57:36.106484 11778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:57:36.107554 11792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:36.107757 11778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:57:36.107856 11778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "6b9a0224831e43b598499664f08de4cb"
format_stamp: "Formatted at 2026-06-20 22:57:36 on dist-test-slave-rdl9"
server_key: "14c7ecfe7829db1a61675d1c68863542"
server_key_iv: "654da15793f17e3460abe17805ca8708"
server_key_version: "encryptionkey@0"
I20260620 22:57:36.107928 11778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:36.111831 11778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:36.112128 11778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:36.112262 11778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:36.112565 11778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:36.112885 11778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:36.113274 11778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:36.113345 11778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:36.113365 11778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:36.120716 11778 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:34481
I20260620 22:57:36.120765 11905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:34481 every 8 connection(s)
I20260620 22:57:36.121189 11778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:57:36.124944 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:36.122157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:59753 (local address 127.1.166.62:43193)
0620 22:57:36.122224 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:36.122226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:36.122668 (+ 442us) server_negotiation.cc:407] Connection header received
0620 22:57:36.123293 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:36.123295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:36.123302 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:36.123389 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:36.124694 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:36.124695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:36.124696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:57:36.124700 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:36.124716 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:57:36.124720 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:36.124789 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:36.124866 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 22:57:36.124881 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:57:36.125159 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:36.122046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:43193 (local address 127.1.166.3:59753)
0620 22:57:36.122511 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:36.122558 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:36.123157 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:36.123465 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:36.123469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:36.123480 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:36.124478 (+ 998us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:57:36.124479 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:36.124521 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:57:36.124565 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:36.124793 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:36.124797 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:36.124801 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:57:36.124861 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:57:36.124925 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":170,"threads_started":1}
I20260620 22:57:36.125671 11906 heartbeater.cc:347] Connected to a master server at 127.1.166.62:43193
I20260620 22:57:36.125782 11906 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:36.125942 11906 heartbeater.cc:511] Master 127.1.166.62:43193 requested a full tablet report, sending...
I20260620 22:57:36.126322 11455 ts_manager.cc:194] Registered new tserver with Master: 6b9a0224831e43b598499664f08de4cb (127.1.166.3:34481)
I20260620 22:57:36.128018 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 11778
I20260620 22:57:36.128108 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:57:36.128284 1688 external_mini_cluster.cc:1428] Setting key 3eedc6d45203f1304b4d773642ac1f68
I20260620 22:57:36.129056 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:57:36.130836 11496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:57:36.130386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.1.166.62:43193)
0620 22:57:36.130444 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:36.130446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:36.130573 (+ 127us) server_negotiation.cc:407] Connection header received
0620 22:57:36.130614 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:36.130615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:36.130620 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:36.130665 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:36.130771 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50854: BlockingRecv error: recv got EOF from 127.0.0.1:50854 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:57:36.130887 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 11516
I20260620 22:57:36.135551 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 11647
I20260620 22:57:36.139685 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 11778
I20260620 22:57:36.143553 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 11425
2026-06-20T22:57:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (559 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:57:36 dist-test-slave-rdl9 krb5kdc[11916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:57:36 dist-test-slave-rdl9 krb5kdc[11916](info): set up 2 sockets
Jun 20 22:57:36 dist-test-slave-rdl9 krb5kdc[11916](info): commencing operation
krb5kdc: starting...
W20260620 22:57:37.110626 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:57:37.233171 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:39493
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:39493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:37.319897 11929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:37.320164 11929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:37.320204 11929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:37.322114 11929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:57:37.322175 11929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:37.322194 11929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:37.322211 11929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:57:37.322227 11929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:57:37.324178 11929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:39493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:39493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:37.324498 11929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:37.324728 11929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:37.330026 11937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:37.330026 11935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:37.330030 11934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:37.330400 11929 server_base.cc:1034] running on GCE node
I20260620 22:57:37.330684 11929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:37.330976 11929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:37.332144 11929 hybrid_clock.cc:648] HybridClock initialized: now 1781996257332126 us; error 38 us; skew 500 ppm
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:37.335234 11929 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:57:37.335749 11929 webserver.cc:460] Webserver started at http://127.1.166.62:41793/ using document root <none> and password file <none>
I20260620 22:57:37.336110 11929 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:37.336184 11929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:37.336318 11929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:37.337663 11929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "82dc7c4d325b4c97861c977793f6e34a"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "69ccda48364a32468a2e547dd58ac679"
server_key_iv: "ccadd418da21f22948c0b9595889f9de"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.338186 11929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "82dc7c4d325b4c97861c977793f6e34a"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "69ccda48364a32468a2e547dd58ac679"
server_key_iv: "ccadd418da21f22948c0b9595889f9de"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.339663 11929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:37.340538 11944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.340816 11929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:37.340961 11929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "82dc7c4d325b4c97861c977793f6e34a"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "69ccda48364a32468a2e547dd58ac679"
server_key_iv: "ccadd418da21f22948c0b9595889f9de"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.341152 11929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:37.353951 11929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:37.355940 11929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:37.356093 11929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:37.360047 11929 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:39493
I20260620 22:57:37.360077 11996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:39493 every 8 connection(s)
I20260620 22:57:37.360415 11929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:57:37.361341 11997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:37.363656 11997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a: Bootstrap starting.
I20260620 22:57:37.364230 11997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:37.364497 11997 log.cc:824] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:37.365216 11997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a: No bootstrap required, opened a new log
I20260620 22:57:37.366863 11997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dc7c4d325b4c97861c977793f6e34a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39493 } }
I20260620 22:57:37.367013 11997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:37.367036 11997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dc7c4d325b4c97861c977793f6e34a, State: Initialized, Role: FOLLOWER
I20260620 22:57:37.367169 11997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dc7c4d325b4c97861c977793f6e34a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39493 } }
I20260620 22:57:37.367249 11997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:57:37.367275 11997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:57:37.367307 11997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:37.368175 11997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dc7c4d325b4c97861c977793f6e34a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39493 } }
I20260620 22:57:37.368273 11997 leader_election.cc:302] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82dc7c4d325b4c97861c977793f6e34a; no voters:
I20260620 22:57:37.368448 11997 leader_election.cc:288] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:57:37.368553 12000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:37.368690 11997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:57:37.368706 12000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [term 1 LEADER]: Becoming Leader. State: Replica: 82dc7c4d325b4c97861c977793f6e34a, State: Running, Role: LEADER
I20260620 22:57:37.368808 12000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dc7c4d325b4c97861c977793f6e34a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39493 } }
I20260620 22:57:37.369416 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 11929
I20260620 22:57:37.369513 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:57:37.369719 1688 external_mini_cluster.cc:1428] Setting key 43e6f0621c60186ca0047e57ffa0ec53
I20260620 22:57:37.369966 12002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82dc7c4d325b4c97861c977793f6e34a. Latest consensus state: current_term: 1 leader_uuid: "82dc7c4d325b4c97861c977793f6e34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dc7c4d325b4c97861c977793f6e34a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39493 } } }
I20260620 22:57:37.370520 12002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [sys.catalog]: This master's current role is: LEADER
I20260620 22:57:37.369177 12001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82dc7c4d325b4c97861c977793f6e34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dc7c4d325b4c97861c977793f6e34a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39493 } } }
W20260620 22:57:37.370627 12015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 22:57:37.370682 12001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a [sys.catalog]: This master's current role is: LEADER
W20260620 22:57:37.370688 12015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:57:37.370787 12019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:57:37.370966 12019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:57:37.372512 12019 catalog_manager.cc:1347] Generated new cluster ID: 1e54f84a1da141f19f5b628cfd4560cf
I20260620 22:57:37.372568 12019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:57:37.382040 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.370217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.1.166.62:39493)
0620 22:57:37.370778 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:37.370790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:37.370824 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:57:37.370958 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:37.370965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:37.371028 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:37.371298 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:37.371988 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.373318 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.374464 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.375109 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.375207 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.378421 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:37.378426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:37.378427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:37.378441 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:37.380402 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.380794 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.380797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.380798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.380932 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.381102 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.381103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.381103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.381423 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:37.381575 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:37.381655 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 22:57:37.381797 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":161,"threads_started":1}
I20260620 22:57:37.395339 12019 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:57:37.395902 12019 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:57:37.399657 12019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a: Generated new TSK 0
I20260620 22:57:37.399840 12019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:57:37.425369 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39493
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:37.508529 12023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:37.508778 12023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:37.508805 12023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:37.510139 12023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:37.510206 12023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:37.510236 12023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:57:37.512137 12023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:37.512430 12023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:37.512725 12023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:37.518158 12031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:37.518129 12028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:37.518235 12029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:37.518687 12023 server_base.cc:1034] running on GCE node
I20260620 22:57:37.518937 12023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:37.519235 12023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:37.520430 12023 hybrid_clock.cc:648] HybridClock initialized: now 1781996257520410 us; error 41 us; skew 500 ppm
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:37.523990 12023 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:57:37.524518 12023 webserver.cc:460] Webserver started at http://127.1.166.1:40795/ using document root <none> and password file <none>
I20260620 22:57:37.524729 12023 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:37.524776 12023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:37.524905 12023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:37.525949 12023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "27b6c65f460f4741acc44a78bf16dd2a"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "77c39789455bc04642fb0dbd4e55a19d"
server_key_iv: "6a26ce86d2ae10e8f1d362a58dbe85f4"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.526273 12023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "27b6c65f460f4741acc44a78bf16dd2a"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "77c39789455bc04642fb0dbd4e55a19d"
server_key_iv: "6a26ce86d2ae10e8f1d362a58dbe85f4"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.527565 12023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:37.528366 12038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.528554 12023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:37.528637 12023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "27b6c65f460f4741acc44a78bf16dd2a"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "77c39789455bc04642fb0dbd4e55a19d"
server_key_iv: "6a26ce86d2ae10e8f1d362a58dbe85f4"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.528700 12023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:37.552596 12023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:37.554841 12023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:37.554989 12023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:37.555215 12023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:37.555541 12023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:37.555573 12023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.555603 12023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:37.555621 12023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.561651 12023 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:33117
I20260620 22:57:37.561705 12151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:33117 every 8 connection(s)
I20260620 22:57:37.562142 12023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:57:37.570536 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 12023
I20260620 22:57:37.570657 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:57:37.570854 1688 external_mini_cluster.cc:1428] Setting key 5de9bda36f71ea6c68d12797647f8bb7
I20260620 22:57:37.573148 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.562967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:59559 (local address 127.1.166.62:39493)
0620 22:57:37.563025 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:37.563026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:37.563182 (+ 156us) server_negotiation.cc:407] Connection header received
0620 22:57:37.563710 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:37.563712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:37.563730 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:37.563793 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:37.564784 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.565309 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.566633 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.566844 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.566907 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.569268 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:37.569270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:37.569270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:37.569279 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:37.571423 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.572047 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.572049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.572050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.572125 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.572418 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.572419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.572419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.572567 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:37.572636 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:37.572962 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 22:57:37.573056 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:57:37.573623 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.562867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39493 (local address 127.1.166.1:59559)
0620 22:57:37.563028 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:37.563073 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:37.563589 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:37.563857 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:37.563860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:37.564459 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:37.564664 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.564669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.565555 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.565556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.566504 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.566507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.566951 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.566953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.567079 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.567610 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:37.567620 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:37.569135 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:37.571522 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:37.571539 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:37.571542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:37.571914 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:37.572213 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:37.572214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:37.572215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:37.572345 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:37.572656 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:37.572662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:37.572828 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 22:57:37.572931 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 22:57:37.573088 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260620 22:57:37.574443 12152 heartbeater.cc:347] Connected to a master server at 127.1.166.62:39493
I20260620 22:57:37.574554 12152 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:37.574747 12152 heartbeater.cc:511] Master 127.1.166.62:39493 requested a full tablet report, sending...
I20260620 22:57:37.575299 11961 ts_manager.cc:194] Registered new tserver with Master: 27b6c65f460f4741acc44a78bf16dd2a (127.1.166.1:33117)
I20260620 22:57:37.575994 11961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:59559
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:57:37.597483 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39493
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:37.679425 12157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:37.679665 12157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:37.679700 12157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:37.681021 12157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:37.681090 12157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:37.681131 12157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:57:37.683296 12157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:37.683847 12157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:37.684177 12157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:37.690138 12162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:37.690212 12157 server_base.cc:1034] running on GCE node
W20260620 22:57:37.690131 12165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:37.690131 12163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:37.690591 12157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:37.690874 12157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:37.692106 12157 hybrid_clock.cc:648] HybridClock initialized: now 1781996257692091 us; error 29 us; skew 500 ppm
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:37.695515 12157 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:57:37.696272 12157 webserver.cc:460] Webserver started at http://127.1.166.2:43981/ using document root <none> and password file <none>
I20260620 22:57:37.696518 12157 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:37.696573 12157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:37.696705 12157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:37.697849 12157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "870bac33daa6480daf2715b1c6da96f8"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "86c557e8e6bd3441844da0813fcf63bd"
server_key_iv: "8678f2dfb25e20c531ad1a13d9d4d409"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.698570 12157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "870bac33daa6480daf2715b1c6da96f8"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "86c557e8e6bd3441844da0813fcf63bd"
server_key_iv: "8678f2dfb25e20c531ad1a13d9d4d409"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.700390 12157 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 22:57:37.701501 12172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.701731 12157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:37.701830 12157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "870bac33daa6480daf2715b1c6da96f8"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "86c557e8e6bd3441844da0813fcf63bd"
server_key_iv: "8678f2dfb25e20c531ad1a13d9d4d409"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.701921 12157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:37.711089 12157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:37.713797 12157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:37.714004 12157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:37.714257 12157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:37.714666 12157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:37.714700 12157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.714723 12157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:37.714746 12157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.721239 12157 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:45387
I20260620 22:57:37.721283 12285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:45387 every 8 connection(s)
I20260620 22:57:37.721602 12157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:57:37.722702 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 12157
I20260620 22:57:37.722805 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:57:37.722995 1688 external_mini_cluster.cc:1428] Setting key acef7dc2cc971e6bae678aab15e54997
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
I20260620 22:57:37.732645 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.722530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:40053 (local address 127.1.166.62:39493)
0620 22:57:37.722707 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:37.722710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:37.722849 (+ 139us) server_negotiation.cc:407] Connection header received
0620 22:57:37.723471 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:37.723472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:37.723480 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:37.723524 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:37.724569 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.725174 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.726637 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.726835 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.726901 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.729522 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:37.729523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:37.729524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:37.729531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:37.730932 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.731362 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.731363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.731364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.731423 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.731651 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.731651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.731652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.731790 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:37.731850 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:37.732454 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 22:57:37.732572 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 22:57:37.732877 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.722381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39493 (local address 127.1.166.2:40053)
0620 22:57:37.722692 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:37.722750 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:37.723384 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:37.723601 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:37.723605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:37.724249 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:37.724459 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.724464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.725262 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.725263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.726534 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.726538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.726933 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.726934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.727082 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.727650 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:37.727660 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:37.729396 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:37.731029 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:37.731038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:37.731040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:37.731297 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:37.731482 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:37.731483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:37.731483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:37.731585 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:37.731878 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:37.731884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:37.732162 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 22:57:37.732323 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 22:57:37.732637 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
I20260620 22:57:37.733418 12286 heartbeater.cc:347] Connected to a master server at 127.1.166.62:39493
I20260620 22:57:37.733542 12286 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:37.733736 12286 heartbeater.cc:511] Master 127.1.166.62:39493 requested a full tablet report, sending...
I20260620 22:57:37.734243 11961 ts_manager.cc:194] Registered new tserver with Master: 870bac33daa6480daf2715b1c6da96f8 (127.1.166.2:45387)
I20260620 22:57:37.734774 11961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:40053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:57:37.746982 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39493
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:57:37.830906 12291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:57:37.831126 12291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:57:37.831152 12291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:57:37.832315 12291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:57:37.832361 12291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:57:37.832388 12291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:57:37.833890 12291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:57:37.834102 12291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:57:37.834292 12291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:57:37.839203 12297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:37.839229 12296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:57:37.839388 12299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:57:37.839471 12291 server_base.cc:1034] running on GCE node
I20260620 22:57:37.839706 12291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:57:37.839974 12291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:57:37.841143 12291 hybrid_clock.cc:648] HybridClock initialized: now 1781996257841128 us; error 33 us; skew 500 ppm
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:57:37.843946 12291 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:57:37.844465 12291 webserver.cc:460] Webserver started at http://127.1.166.3:41995/ using document root <none> and password file <none>
I20260620 22:57:37.844702 12291 fs_manager.cc:359] Metadata directory not provided
I20260620 22:57:37.844754 12291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:57:37.844884 12291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:57:37.845953 12291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "83be355595a2473cb370f3e085b888ac"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "321272cd193102b94ff93b27421959a8"
server_key_iv: "5a88837915a0970608e24951fabd91f1"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.846292 12291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "83be355595a2473cb370f3e085b888ac"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "321272cd193102b94ff93b27421959a8"
server_key_iv: "5a88837915a0970608e24951fabd91f1"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.847800 12291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:57:37.848991 12306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.849285 12291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:57:37.849397 12291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "83be355595a2473cb370f3e085b888ac"
format_stamp: "Formatted at 2026-06-20 22:57:37 on dist-test-slave-rdl9"
server_key: "321272cd193102b94ff93b27421959a8"
server_key_iv: "5a88837915a0970608e24951fabd91f1"
server_key_version: "encryptionkey@0"
I20260620 22:57:37.849488 12291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:57:37.859566 12291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:57:37.861997 12291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:57:37.862138 12291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:57:37.862442 12291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:57:37.862749 12291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:57:37.862782 12291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.862812 12291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:57:37.862836 12291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:57:37.868422 12291 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:40121
I20260620 22:57:37.868460 12419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:40121 every 8 connection(s)
I20260620 22:57:37.868770 12291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:57:37.872284 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 12291
I20260620 22:57:37.872396 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:57:37.872573 1688 external_mini_cluster.cc:1428] Setting key 183858e7331b289365d3110d68337382
Jun 20 22:57:37 dist-test-slave-rdl9 krb5kdc[11916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996257, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:57:37.880703 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.869714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:57679 (local address 127.1.166.62:39493)
0620 22:57:37.869756 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:37.869758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:37.869769 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:57:37.870630 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:37.870631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:37.870642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:37.870705 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:37.872360 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.873299 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.874397 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.874586 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.874644 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.877263 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:37.877265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:37.877265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:37.877272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:37.879028 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.879445 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.879446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.879447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.879499 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.879968 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.879969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.879970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.880104 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:37.880169 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:37.880443 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 22:57:37.880528 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:57:37.880926 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.869476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39493 (local address 127.1.166.3:57679)
0620 22:57:37.869628 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:37.869669 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:37.870476 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:37.870892 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:37.870904 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:37.871489 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:57:37.871692 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.871696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.873415 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.873417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.874273 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.874275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.874667 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.874669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.874802 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.875322 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:57:37.875331 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:57:37.877135 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:57:37.879124 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:37.879133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:37.879135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:37.879376 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:37.879555 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:57:37.879556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:57:37.879556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:57:37.879715 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:57:37.880169 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:57:37.880177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:57:37.880319 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 22:57:37.880414 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 22:57:37.880670 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260620 22:57:37.881448 12420 heartbeater.cc:347] Connected to a master server at 127.1.166.62:39493
I20260620 22:57:37.881541 12420 heartbeater.cc:464] Registering TS with master...
I20260620 22:57:37.881796 12420 heartbeater.cc:511] Master 127.1.166.62:39493 requested a full tablet report, sending...
I20260620 22:57:37.882325 11961 ts_manager.cc:194] Registered new tserver with Master: 83be355595a2473cb370f3e085b888ac (127.1.166.3:40121)
I20260620 22:57:37.882850 11961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:57679
I20260620 22:57:37.884110 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:57:37.891258 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.885549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.1.166.62:39493)
0620 22:57:37.885597 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:37.885599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:37.885613 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:57:37.885678 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:37.885679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:37.885689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:37.885745 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:57:37.886264 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.886780 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.887483 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.887664 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.887714 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.889010 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:57:37.889011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:57:37.889011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:57:37.889020 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:57:37.890080 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.890482 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.890483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.890484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.890543 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:57:37.890739 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:57:37.890740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:57:37.890740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:57:37.890908 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:57:37.890998 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:37.891124 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 22:57:37.891189 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:57:37.893462 11961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:57:37.895093 11961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:57:37.901657 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.897653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.1.166.2:45387)
0620 22:57:37.897833 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:37.897838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:37.897848 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:57:37.897889 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:37.897891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:37.897905 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:37.897979 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:37.898479 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.899656 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.900799 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.901279 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.901342 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.901404 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:37.901533 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:57:37.901574 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260620 22:57:37.901679 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.897314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45387 (local address 127.0.0.1:59280)
0620 22:57:37.897612 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:37.897624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:37.897693 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:37.898194 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:37.898195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:37.898197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:37.898275 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.898276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.899760 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.899761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.900709 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.900711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.901355 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.901356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.901434 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.901448 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:57:37.901512 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:57:37.901568 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":26,"thread_start_us":48,"threads_started":1}
I20260620 22:57:37.901849 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.897452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.1.166.3:40121)
0620 22:57:37.897833 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:37.897838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:37.897848 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:57:37.897886 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:37.897888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:37.897961 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:37.898062 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:37.898381 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.899073 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.900444 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.901283 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.901354 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.901417 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:37.901722 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 22:57:37.901769 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":69,"threads_started":1}
I20260620 22:57:37.902097 12431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.897114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:33117 (local address 127.0.0.1:55552)
0620 22:57:37.897586 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:37.897624 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:37.897693 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:37.898144 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:37.898147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:37.898150 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:37.898265 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.898267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.899775 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.899776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.900851 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.900855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.901820 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.901821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.901897 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.901900 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:57:37.901929 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 22:57:37.901966 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":78,"thread_start_us":69,"threads_started":1}
I20260620 22:57:37.902300 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.897190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:40121 (local address 127.0.0.1:44026)
0620 22:57:37.897581 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:37.897624 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:37.897693 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:37.898145 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:37.898147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:37.898152 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:37.898280 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.898283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.899168 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.899169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.900361 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:37.900364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.901453 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.901454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:37.901519 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.901523 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:57:37.901568 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:57:37.901612 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":40,"thread_start_us":96,"threads_started":1}
I20260620 22:57:37.902503 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:37.897214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.1.166.1:33117)
0620 22:57:37.897862 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:37.897866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:37.897874 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:57:37.897916 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:37.897918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:37.897932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:37.898358 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:37.898490 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.899636 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.900963 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:37.901736 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:37.902262 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:37.902328 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:37.902382 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 22:57:37.902426 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":79,"threads_started":1}
I20260620 22:57:37.902655 12220 tablet_service.cc:1467] Processing CreateTablet for tablet 9b227fd70d9c43c9ae12f0154501cb1e (DEFAULT_TABLE table=test-table [id=f701bd171d8f48b88a267e9c8841050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:37.903028 12220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b227fd70d9c43c9ae12f0154501cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:37.903051 12354 tablet_service.cc:1467] Processing CreateTablet for tablet 9b227fd70d9c43c9ae12f0154501cb1e (DEFAULT_TABLE table=test-table [id=f701bd171d8f48b88a267e9c8841050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:37.903057 12086 tablet_service.cc:1467] Processing CreateTablet for tablet 9b227fd70d9c43c9ae12f0154501cb1e (DEFAULT_TABLE table=test-table [id=f701bd171d8f48b88a267e9c8841050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:57:37.903354 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b227fd70d9c43c9ae12f0154501cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:37.903354 12086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b227fd70d9c43c9ae12f0154501cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:57:37.906117 12434 tablet_bootstrap.cc:492] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Bootstrap starting.
I20260620 22:57:37.906778 12435 tablet_bootstrap.cc:492] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a: Bootstrap starting.
I20260620 22:57:37.907055 12434 tablet_bootstrap.cc:654] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:37.907449 12435 tablet_bootstrap.cc:654] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:37.907698 12434 log.cc:824] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:37.907979 12435 log.cc:824] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:37.908311 12436 tablet_bootstrap.cc:492] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Bootstrap starting.
I20260620 22:57:37.908928 12434 tablet_bootstrap.cc:492] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: No bootstrap required, opened a new log
I20260620 22:57:37.909010 12434 ts_tablet_manager.cc:1397] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 22:57:37.909085 12436 tablet_bootstrap.cc:654] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Neither blocks nor log segments found. Creating new log.
I20260620 22:57:37.909628 12436 log.cc:824] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Log is configured to *not* fsync() on all Append() calls
I20260620 22:57:37.910197 12435 tablet_bootstrap.cc:492] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a: No bootstrap required, opened a new log
I20260620 22:57:37.910272 12435 ts_tablet_manager.cc:1397] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 22:57:37.910503 12436 tablet_bootstrap.cc:492] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: No bootstrap required, opened a new log
I20260620 22:57:37.910571 12436 ts_tablet_manager.cc:1397] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:57:37.912034 12436 raft_consensus.cc:348] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:37.912199 12436 raft_consensus.cc:374] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:37.912235 12436 raft_consensus.cc:729] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83be355595a2473cb370f3e085b888ac, State: Initialized, Role: FOLLOWER
I20260620 22:57:37.912359 12436 consensus_queue.cc:260] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:37.912451 12434 raft_consensus.cc:348] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:37.912678 12436 ts_tablet_manager.cc:1428] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:57:37.912739 12434 raft_consensus.cc:374] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:37.912778 12434 raft_consensus.cc:729] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870bac33daa6480daf2715b1c6da96f8, State: Initialized, Role: FOLLOWER
I20260620 22:57:37.912943 12420 heartbeater.cc:503] Master 127.1.166.62:39493 was elected leader, sending a full tablet report...
I20260620 22:57:37.912962 12434 consensus_queue.cc:260] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:37.913385 12434 ts_tablet_manager.cc:1428] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 22:57:37.913589 12286 heartbeater.cc:503] Master 127.1.166.62:39493 was elected leader, sending a full tablet report...
I20260620 22:57:37.913571 12435 raft_consensus.cc:348] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:37.913741 12435 raft_consensus.cc:374] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:57:37.913774 12435 raft_consensus.cc:729] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b6c65f460f4741acc44a78bf16dd2a, State: Initialized, Role: FOLLOWER
I20260620 22:57:37.913883 12435 consensus_queue.cc:260] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:37.914312 12435 ts_tablet_manager.cc:1428] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260620 22:57:37.914390 12152 heartbeater.cc:503] Master 127.1.166.62:39493 was elected leader, sending a full tablet report...
W20260620 22:57:37.972322 12287 tablet.cc:2369] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:57:38.063084 12153 tablet.cc:2369] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:57:38.087888 12442 raft_consensus.cc:482] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:57:38.088044 12442 raft_consensus.cc:504] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:38.088393 12442 leader_election.cc:288] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 870bac33daa6480daf2715b1c6da96f8 (127.1.166.2:45387), 83be355595a2473cb370f3e085b888ac (127.1.166.3:40121)
I20260620 22:57:38.093189 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:38.088679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:40493 (local address 127.1.166.2:45387)
0620 22:57:38.088730 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:38.088732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:38.088869 (+ 137us) server_negotiation.cc:407] Connection header received
0620 22:57:38.088905 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:38.088906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:38.088921 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:38.088983 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:38.089249 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.090078 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.091988 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.092678 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.092801 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:38.092817 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:38.093037 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 22:57:38.093091 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:57:38.093189 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:38.088693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:44159 (local address 127.1.166.3:40121)
0620 22:57:38.088740 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:38.088742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:38.088869 (+ 127us) server_negotiation.cc:407] Connection header received
0620 22:57:38.088905 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:38.088906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:38.088919 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:38.088979 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:57:38.089249 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.090078 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.091988 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.092674 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.092801 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:38.092817 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:38.093040 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 22:57:38.093092 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:57:38.093235 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:38.088615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45387 (local address 127.1.166.1:40493)
0620 22:57:38.088746 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:38.088754 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:38.088815 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:38.089020 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:38.089020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:38.089023 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:38.089144 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:38.089148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.090207 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.090208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:38.091792 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:38.091796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.092832 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.092834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:38.092935 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:38.092945 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 22:57:38.093021 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:57:38.093073 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":132,"thread_start_us":59,"threads_started":1}
I20260620 22:57:38.093233 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:38.088615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:40121 (local address 127.1.166.1:44159)
0620 22:57:38.088735 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:38.088746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:57:38.088815 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:57:38.089019 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:57:38.089020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:57:38.089025 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:57:38.089144 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:38.089147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.090207 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.090208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:38.091791 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:57:38.091796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.092833 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.092834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:57:38.092935 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:38.092944 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 22:57:38.093027 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:57:38.093073 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":11,"thread_start_us":60,"threads_started":1}
I20260620 22:57:38.093710 12240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b227fd70d9c43c9ae12f0154501cb1e" candidate_uuid: "27b6c65f460f4741acc44a78bf16dd2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870bac33daa6480daf2715b1c6da96f8" is_pre_election: true
I20260620 22:57:38.093880 12374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b227fd70d9c43c9ae12f0154501cb1e" candidate_uuid: "27b6c65f460f4741acc44a78bf16dd2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83be355595a2473cb370f3e085b888ac" is_pre_election: true
I20260620 22:57:38.093946 12240 raft_consensus.cc:2454] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b6c65f460f4741acc44a78bf16dd2a in term 0.
I20260620 22:57:38.094035 12374 raft_consensus.cc:2454] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b6c65f460f4741acc44a78bf16dd2a in term 0.
I20260620 22:57:38.094249 12042 leader_election.cc:302] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b6c65f460f4741acc44a78bf16dd2a, 870bac33daa6480daf2715b1c6da96f8; no voters:
I20260620 22:57:38.094435 12442 raft_consensus.cc:2781] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:57:38.094499 12442 raft_consensus.cc:482] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:57:38.094516 12442 raft_consensus.cc:3037] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:38.095217 12442 raft_consensus.cc:504] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:38.095345 12442 leader_election.cc:288] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [CANDIDATE]: Term 1 election: Requested vote from peers 870bac33daa6480daf2715b1c6da96f8 (127.1.166.2:45387), 83be355595a2473cb370f3e085b888ac (127.1.166.3:40121)
I20260620 22:57:38.095549 12374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b227fd70d9c43c9ae12f0154501cb1e" candidate_uuid: "27b6c65f460f4741acc44a78bf16dd2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83be355595a2473cb370f3e085b888ac"
I20260620 22:57:38.095641 12374 raft_consensus.cc:3037] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:38.095710 12240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b227fd70d9c43c9ae12f0154501cb1e" candidate_uuid: "27b6c65f460f4741acc44a78bf16dd2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870bac33daa6480daf2715b1c6da96f8"
I20260620 22:57:38.095813 12240 raft_consensus.cc:3037] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:57:38.096378 12374 raft_consensus.cc:2454] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b6c65f460f4741acc44a78bf16dd2a in term 1.
I20260620 22:57:38.096581 12240 raft_consensus.cc:2454] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b6c65f460f4741acc44a78bf16dd2a in term 1.
I20260620 22:57:38.096583 12040 leader_election.cc:302] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b6c65f460f4741acc44a78bf16dd2a, 83be355595a2473cb370f3e085b888ac; no voters:
I20260620 22:57:38.096669 12442 raft_consensus.cc:2781] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:57:38.096822 12442 raft_consensus.cc:686] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 1 LEADER]: Becoming Leader. State: Replica: 27b6c65f460f4741acc44a78bf16dd2a, State: Running, Role: LEADER
I20260620 22:57:38.096910 12442 consensus_queue.cc:237] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } }
I20260620 22:57:38.097720 11961 catalog_manager.cc:5495] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 27b6c65f460f4741acc44a78bf16dd2a (127.1.166.1). New cstate: current_term: 1 leader_uuid: "27b6c65f460f4741acc44a78bf16dd2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27b6c65f460f4741acc44a78bf16dd2a" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 } health_report { overall_health: UNKNOWN } } }
W20260620 22:57:38.119344 12421 tablet.cc:2369] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:57:38.133911 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:38.131623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.1.166.1:33117)
0620 22:57:38.131681 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:57:38.131683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:57:38.131692 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:57:38.131728 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:57:38.131729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:57:38.131748 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:57:38.131806 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:57:38.131931 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.132401 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.133183 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:57:38.133385 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:57:38.133450 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:57:38.133613 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:57:38.133720 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:57:38.133781 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:57:38.133820 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 22:57:38.133845 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:57:38.135792 12240 raft_consensus.cc:1261] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 1 FOLLOWER]: Refusing update from remote peer 27b6c65f460f4741acc44a78bf16dd2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:38.135797 12374 raft_consensus.cc:1261] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 1 FOLLOWER]: Refusing update from remote peer 27b6c65f460f4741acc44a78bf16dd2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:57:38.136142 12445 consensus_queue.cc:1035] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83be355595a2473cb370f3e085b888ac" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:38.136277 12442 consensus_queue.cc:1035] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "870bac33daa6480daf2715b1c6da96f8" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:57:38.141234 11961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:38.141340 11961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:57:38.142122 11961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82dc7c4d325b4c97861c977793f6e34a: Sending DeleteTablet for 3 replicas of tablet 9b227fd70d9c43c9ae12f0154501cb1e
I20260620 22:57:38.142454 12086 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b227fd70d9c43c9ae12f0154501cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:38 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:55552
I20260620 22:57:38.142499 12354 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b227fd70d9c43c9ae12f0154501cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:38 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:44026
I20260620 22:57:38.142518 12220 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b227fd70d9c43c9ae12f0154501cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:57:38 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:59280
I20260620 22:57:38.142676 12458 tablet_replica.cc:330] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a: stopping tablet replica
I20260620 22:57:38.142771 12458 raft_consensus.cc:2229] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:57:38.142905 12458 raft_consensus.cc:2258] T 9b227fd70d9c43c9ae12f0154501cb1e P 27b6c65f460f4741acc44a78bf16dd2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:38.143080 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 12023
I20260620 22:57:38.145239 12460 tablet_replica.cc:330] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: stopping tablet replica
I20260620 22:57:38.145392 12460 raft_consensus.cc:2229] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:38.145509 12460 raft_consensus.cc:2258] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:38.145526 12459 tablet_replica.cc:330] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: stopping tablet replica
I20260620 22:57:38.145668 12459 raft_consensus.cc:2229] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:57:38.145782 12459 raft_consensus.cc:2258] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:57:38.146072 12460 ts_tablet_manager.cc:1905] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:38.146203 12459 ts_tablet_manager.cc:1905] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:57:38.147794 12460 ts_tablet_manager.cc:1918] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:57:38.147794 12459 ts_tablet_manager.cc:1918] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:57:38.147881 12460 log.cc:1196] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/9b227fd70d9c43c9ae12f0154501cb1e
I20260620 22:57:38.147881 12459 log.cc:1196] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/9b227fd70d9c43c9ae12f0154501cb1e
I20260620 22:57:38.148159 12459 ts_tablet_manager.cc:1939] T 9b227fd70d9c43c9ae12f0154501cb1e P 83be355595a2473cb370f3e085b888ac: Deleting consensus metadata
I20260620 22:57:38.148159 12460 ts_tablet_manager.cc:1939] T 9b227fd70d9c43c9ae12f0154501cb1e P 870bac33daa6480daf2715b1c6da96f8: Deleting consensus metadata
W20260620 22:57:38.148387 11948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:33117 (error 108)
I20260620 22:57:38.148571 11946 catalog_manager.cc:4841] TS 83be355595a2473cb370f3e085b888ac (127.1.166.3:40121): tablet 9b227fd70d9c43c9ae12f0154501cb1e (table test-table [id=f701bd171d8f48b88a267e9c8841050d]) successfully deleted
I20260620 22:57:38.148730 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 12157
I20260620 22:57:38.148785 11948 catalog_manager.cc:4841] TS 870bac33daa6480daf2715b1c6da96f8 (127.1.166.2:45387): tablet 9b227fd70d9c43c9ae12f0154501cb1e (table test-table [id=f701bd171d8f48b88a267e9c8841050d]) successfully deleted
I20260620 22:57:38.148861 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:57:38.148738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:33117 (local address 127.0.0.1:55580)
0620 22:57:38.148793 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:57:38.148830 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:33117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 22:57:38.149017 11948 catalog_manager.cc:4584] TS 27b6c65f460f4741acc44a78bf16dd2a (127.1.166.1:33117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b227fd70d9c43c9ae12f0154501cb1e: Network error: Client connection negotiation failed: client connection to 127.1.166.1:33117: connect: Connection refused (error 111)
I20260620 22:57:38.153656 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 12291
I20260620 22:57:38.157997 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 11929
2026-06-20T22:57:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2015 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:57:38 dist-test-slave-rdl9 krb5kdc[12464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:57:38 dist-test-slave-rdl9 krb5kdc[12464](info): set up 2 sockets
Jun 20 22:57:38 dist-test-slave-rdl9 krb5kdc[12464](info): commencing operation
krb5kdc: starting...
W20260620 22:57:39.134719 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:57:39 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:57:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/rangeradmin_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/rangeradmin_127.1.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/rangerlookup_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/rangerlookup_127.1.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/HTTP_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/HTTP_127.1.166.20@KRBTEST.COM.keytab.
I20260620 22:57:39.314481 1688 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskE9BVnS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/postgres -l logfile start
2026-06-20 22:57:41.754 UTC [12494] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 22:57:41.754 UTC [12494] LOG: listening on IPv4 address "127.1.166.20", port 39657
2026-06-20 22:57:41.758 UTC [12494] LOG: listening on Unix socket "/tmp/.s.PGSQL.39657"
2026-06-20 22:57:41.785 UTC [12497] LOG: database system was shut down at 2026-06-20 22:57:40 UTC
2026-06-20 22:57:41.788 UTC [12494] LOG: database system is ready to accept connections
I20260620 22:57:42.690390 1688 mini_postgres.cc:96] Postgres bound to 39657
2026-06-20 22:57:42.696 UTC [12505] FATAL: database "slave" does not exist
127.1.166.20:39657 - accepting connections
I20260620 22:57:42.697764 1688 mini_ranger.cc:161] Starting Ranger...
I20260620 22:57:42.714425 1688 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 22:57:43.175414 1688 mini_ranger.cc:87] Created ranger Postgres database
I20260620 22:57:43.175572 1688 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-admin
2026-06-20 22:57:43,454 [I] DB FLAVOR :POSTGRES
2026-06-20 22:57:43,454 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:57:43,454 [I] Checking connection..
2026-06-20 22:57:43,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 22:57:43,724 [I] Checking connection passed.
2026-06-20 22:57:43,724 [I] --------- Verifying version history table ---------
2026-06-20 22:57:43,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:57:44,008 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:57:44,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:57:44,292 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:57:44,292 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 22:57:44,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 22:57:44.551 UTC [12569] WARNING: there is no transaction in progress
2026-06-20 22:57:44,558 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 22:57:44,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:57:44,821 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 22:57:44,821 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:57:44,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 22:57:45,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 22:57:45,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rdl9.c.gcp-upstream.internal','N') ;"
2026-06-20 22:57:45,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:57:45,940 [I] Table x_portal_user does not exist in database ranger
2026-06-20 22:57:45,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:57:46,231 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 22:57:46,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:57:46,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:57:46,760 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 22:57:46,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 22:57:47.813 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:47.832 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:47.851 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:47.869 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:47.888 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:47.907 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:47.971 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:47.980 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:47.997 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:48.010 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:48.023 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:48.040 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:48.050 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:48.057 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:48.065 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:48.071 UTC [12677] WARNING: there is no transaction in progress
2026-06-20 22:57:48,813 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 22:57:48,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 22:57:49,090 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 22:57:49,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:57:49,378 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 22:57:49,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:57:49,670 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 22:57:49,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:57:49,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:57:50,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rdl9.c.gcp-upstream.internal';"
2026-06-20 22:57:50,491 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 22:57:50,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 22:57:50,763 [I] DB_PATCHES have already been applied
I20260620 22:57:50.770062 1688 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-admin:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskE9BVnS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 22:57:50.770164 1688 mini_ranger.cc:193] Using host: 127.1.166.20
I20260620 22:57:50.773483 1688 mini_ranger.cc:227] Ranger admin URL: http://127.1.166.20:48535
Jun 20, 2026 10:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 10:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 10:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 10:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 10:57:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 22:57:52.909634 1688 mini_ranger.cc:160] Time spent starting Ranger: real 10.212s user 0.001s sys 0.011s
Jun 20, 2026 10:58:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 10:58:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 10:58:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 10:58:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 10:58:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 22:58:19.048192 1688 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/rangerkms_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/rangerkms_127.1.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.166.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.166.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.166.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/HTTP_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.166.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/HTTP_127.1.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 22:58:19 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996299, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 22:58:19.132714 1688 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 22:58:19.143093 1688 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 22:58:19.840210 1688 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 22:58:19.840314 1688 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms
I20260620 22:58:19.840332 1688 mini_ranger_kms.cc:227] Using postgres at 127.1.166.20:39657
2026-06-20 22:58:20,086 [I] DB FLAVOR :POSTGRES
2026-06-20 22:58:20,087 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:58:20,087 [I] Checking connection
2026-06-20 22:58:20,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 22:58:20,392 [I] connection success
2026-06-20 22:58:20,393 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 22:58:20,393 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 22:58:20,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 22:58:20,700 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 22:58:20,700 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:58:20,700 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 22:58:20,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:39657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 22:58:20,988 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 22:58:21.571059 1688 mini_ranger_kms.cc:310] Created kms service
I20260620 22:58:21.858800 1688 mini_ranger_kms.cc:326] Created kudu user
I20260620 22:58:21.910908 1688 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 22:58:21 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996299, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:58:22.383256 1688 mini_ranger_kms.cc:384] Added ranger policy
I20260620 22:58:22.383395 1688 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms:/tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 22:58:22.383419 1688 mini_ranger_kms.cc:240] Using host: 127.1.166.20
I20260620 22:58:22.386886 1688 mini_ranger_kms.cc:276] Ranger KMS PID: 12859
I20260620 22:58:22.387027 1688 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.166.20:57211
Jun 20, 2026 10:58:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:58:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:58:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 10:58:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 10:58:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 10:58:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 22:58:24.432118 1688 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.299s user 0.003s sys 0.223s
I20260620 22:58:24.432219 1688 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 22:58:24.432299 1688 mini_ranger_kms.cc:401] 127.1.166.20:57211/kms/v1/keys
Jun 20 22:58:24 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996304, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.166.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 22:58:25 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996304, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.166.20@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
Jun 20 22:58:25 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996304, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.166.20@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
Jun 20 22:58:27 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:58:27.979494 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:36013
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:36013
--ranger_config_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:57211/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:58:28.065364 12909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:58:28.065618 12909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:58:28.065640 12909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:58:28.067030 12909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:58:28.067103 12909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:58:28.067117 12909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:58:28.067128 12909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:58:28.067140 12909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:58:28.068895 12909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.166.20:57211/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:36013
--ranger_config_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:58:28.069149 12909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:58:28.069380 12909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:58:28.074689 12917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:58:28.074689 12915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:58:28.074766 12914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:58:28.074808 12909 server_base.cc:1034] running on GCE node
I20260620 22:58:28.075273 12909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:58:28.075603 12909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:58:28.076761 12909 hybrid_clock.cc:648] HybridClock initialized: now 1781996308076733 us; error 44 us; skew 500 ppm
Jun 20 22:58:28 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996308, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:58:28.079627 12909 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:58:28.080143 12909 webserver.cc:460] Webserver started at http://127.1.166.62:46509/ using document root <none> and password file <none>
I20260620 22:58:28.080349 12909 fs_manager.cc:359] Metadata directory not provided
I20260620 22:58:28.080394 12909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:58:28.080528 12909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:58:28 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996308, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:58:28.147821 12909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "5476aa512ed3467c8939c5888d276839"
format_stamp: "Formatted at 2026-06-20 22:58:28 on dist-test-slave-rdl9"
server_key: "87a20c90e74a1f037a9f533dee00bb75"
server_key_iv: "91521c6990bc765dd93eaa11a2052540"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:28.148313 12909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "5476aa512ed3467c8939c5888d276839"
format_stamp: "Formatted at 2026-06-20 22:58:28 on dist-test-slave-rdl9"
server_key: "87a20c90e74a1f037a9f533dee00bb75"
server_key_iv: "91521c6990bc765dd93eaa11a2052540"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:28.149731 12909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:58:28.166303 12924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:28.166563 12909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:58:28.166661 12909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "5476aa512ed3467c8939c5888d276839"
format_stamp: "Formatted at 2026-06-20 22:58:28 on dist-test-slave-rdl9"
server_key: "87a20c90e74a1f037a9f533dee00bb75"
server_key_iv: "91521c6990bc765dd93eaa11a2052540"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:28.166752 12909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:58:28.202718 12909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:58:28.204596 12909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:58:28.204749 12909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:58:28.209308 12909 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:36013
I20260620 22:58:28.209525 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:36013 every 8 connection(s)
I20260620 22:58:28.209846 12909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:58:28.210891 12977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:58:28.216753 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839: Bootstrap starting.
I20260620 22:58:28.216971 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 12909
I20260620 22:58:28.217178 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:58:28.217393 1688 external_mini_cluster.cc:1428] Setting key ad8826bacd60352950b57917c42a915f
I20260620 22:58:28.217866 12977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839: Neither blocks nor log segments found. Creating new log.
I20260620 22:58:28.218449 12977 log.cc:824] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839: Log is configured to *not* fsync() on all Append() calls
I20260620 22:58:28.219381 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839: No bootstrap required, opened a new log
I20260620 22:58:28.221787 12977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476aa512ed3467c8939c5888d276839" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36013 } }
I20260620 22:58:28.221956 12977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:58:28.221988 12977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5476aa512ed3467c8939c5888d276839, State: Initialized, Role: FOLLOWER
I20260620 22:58:28.222144 12977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476aa512ed3467c8939c5888d276839" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36013 } }
I20260620 22:58:28.222224 12977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:58:28.222257 12977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:58:28.222309 12977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:58:28.223116 12977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476aa512ed3467c8939c5888d276839" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36013 } }
I20260620 22:58:28.223242 12977 leader_election.cc:302] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5476aa512ed3467c8939c5888d276839; no voters:
I20260620 22:58:28.223503 12977 leader_election.cc:288] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:58:28.223655 12982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:58:28.224388 12982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [term 1 LEADER]: Becoming Leader. State: Replica: 5476aa512ed3467c8939c5888d276839, State: Running, Role: LEADER
I20260620 22:58:28.224582 12982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476aa512ed3467c8939c5888d276839" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36013 } }
I20260620 22:58:28.224982 12982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5476aa512ed3467c8939c5888d276839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476aa512ed3467c8939c5888d276839" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36013 } } }
I20260620 22:58:28.225149 12982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [sys.catalog]: This master's current role is: LEADER
I20260620 22:58:28.225224 12983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5476aa512ed3467c8939c5888d276839. Latest consensus state: current_term: 1 leader_uuid: "5476aa512ed3467c8939c5888d276839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476aa512ed3467c8939c5888d276839" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36013 } } }
I20260620 22:58:28.225338 12983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [sys.catalog]: This master's current role is: LEADER
I20260620 22:58:28.225168 12977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:58:28.225416 12986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:58:28.226033 12986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:58:28 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:58:28.227000 12977 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:58:28.237316 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:28.217939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.1.166.62:36013)
0620 22:58:28.218153 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:28.218163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:28.218195 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:58:28.218788 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:28.218798 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:28.218872 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:28.219036 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:58:28.219967 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:28.221379 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:28.222656 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:28.223325 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:28.223513 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:28.227549 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:58:28.227555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:58:28.227555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:58:28.227570 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:58:28.233696 (+ 6126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:28.234250 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:28.234254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:28.234255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:28.234400 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:28.234622 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:28.234623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:28.234623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:28.235009 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:58:28.236624 (+ 1615us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:28.236741 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:58:28.236973 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260620 22:58:28.239459 12998 mvcc.cc:205] Tried to move back new op lower bound from 7299056878545825792 to 7299056878488293376. Current Snapshot: MvccSnapshot[applied={T|T < 7299056878545825792}]
I20260620 22:58:28.244921 12986 catalog_manager.cc:1347] Generated new cluster ID: 4fd06a91c8a243068aeeb6bd36082b0f
I20260620 22:58:28.244998 12986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:58:28.273633 12986 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:58:28.274605 12986 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:58:28.280781 12986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839: Generated new TSK 0
I20260620 22:58:28.281126 12986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:58:29.071525 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:57211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:36013
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:58:29.162931 13022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:58:29.163156 13022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:58:29.163182 13022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:58:29.164444 13022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:58:29.164507 13022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:58:29.164547 13022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:58:29.166217 13022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.166.20:57211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:58:29.166531 13022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:58:29.166772 13022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:58:29.171943 13027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:58:29.172024 13022 server_base.cc:1034] running on GCE node
W20260620 22:58:29.172083 13028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:58:29.171978 13030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:58:29.172470 13022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:58:29.172737 13022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:58:29.173918 13022 hybrid_clock.cc:648] HybridClock initialized: now 1781996309173902 us; error 34 us; skew 500 ppm
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:58:29.177927 13022 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:58:29.178788 13022 webserver.cc:460] Webserver started at http://127.1.166.1:43933/ using document root <none> and password file <none>
I20260620 22:58:29.179111 13022 fs_manager.cc:359] Metadata directory not provided
I20260620 22:58:29.179178 13022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:58:29.179322 13022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:58:29.194435 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "1de084f6d8b547c9b5516af47fdff6bf"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "86de21e14e3b5ac09da89ef4f972c5af"
server_key_iv: "538383beccc51a7fab3510d9df38fea3"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.194911 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "1de084f6d8b547c9b5516af47fdff6bf"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "86de21e14e3b5ac09da89ef4f972c5af"
server_key_iv: "538383beccc51a7fab3510d9df38fea3"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.196786 13022 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 22:58:29.214526 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.214804 13022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260620 22:58:29.214953 13022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "1de084f6d8b547c9b5516af47fdff6bf"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "86de21e14e3b5ac09da89ef4f972c5af"
server_key_iv: "538383beccc51a7fab3510d9df38fea3"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.215142 13022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:58:29.247221 13022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:58:29.249413 13022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:58:29.249650 13022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:58:29.249934 13022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:58:29.250313 13022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:58:29.250378 13022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.250411 13022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:58:29.250427 13022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.257865 13022 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:36771
I20260620 22:58:29.258271 13022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:58:29.258471 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:36771 every 8 connection(s)
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:58:29.267006 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 13022
I20260620 22:58:29.267117 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:58:29.267320 1688 external_mini_cluster.cc:1428] Setting key acf40bcb641170eab782b4ded358ef85
I20260620 22:58:29.270536 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.259345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:36013 (local address 127.1.166.1:49419)
0620 22:58:29.259604 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:29.259650 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:58:29.260326 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:58:29.260629 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:58:29.260634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:58:29.260913 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:58:29.261198 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.261204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.262218 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.262219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.263524 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.263527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.264045 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.264046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.264192 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.264855 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:58:29.264865 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:58:29.266580 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:58:29.268538 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:58:29.268549 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:58:29.268552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:58:29.268914 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:58:29.269142 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:58:29.269143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:58:29.269144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:58:29.269299 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:58:29.269614 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:58:29.269623 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:58:29.269819 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 22:58:29.269936 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 22:58:29.270197 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20260620 22:58:29.270638 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.259384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:49419 (local address 127.1.166.62:36013)
0620 22:58:29.259535 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:29.259537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:29.259759 (+ 222us) server_negotiation.cc:407] Connection header received
0620 22:58:29.260444 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:29.260445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:29.260459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:29.260548 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:58:29.261308 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.262112 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.263621 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.263943 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.264016 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.266708 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:58:29.266710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:58:29.266711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:58:29.266721 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:58:29.268450 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:29.268989 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:29.268990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:29.268990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:29.269077 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:29.269377 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:29.269377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:29.269378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:29.269516 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:58:29.269581 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:29.270434 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 22:58:29.270532 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260620 22:58:29.271410 13153 heartbeater.cc:347] Connected to a master server at 127.1.166.62:36013
I20260620 22:58:29.271514 13153 heartbeater.cc:464] Registering TS with master...
I20260620 22:58:29.271703 13153 heartbeater.cc:511] Master 127.1.166.62:36013 requested a full tablet report, sending...
I20260620 22:58:29.272277 12941 ts_manager.cc:194] Registered new tserver with Master: 1de084f6d8b547c9b5516af47fdff6bf (127.1.166.1:36771)
I20260620 22:58:29.272890 12941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:49419
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:58:29.301535 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:57211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:36013
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:58:29.412254 13159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:58:29.412496 13159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:58:29.412531 13159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:58:29.413780 13159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:58:29.413830 13159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:58:29.413868 13159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:58:29.415618 13159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.166.20:57211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:58:29.415894 13159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:58:29.416133 13159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:58:29.421604 13166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:58:29.421567 13169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:58:29.421902 13167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:58:29.422749 13159 server_base.cc:1034] running on GCE node
I20260620 22:58:29.422971 13159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:58:29.423204 13159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:58:29.430182 13159 hybrid_clock.cc:648] HybridClock initialized: now 1781996309430135 us; error 56 us; skew 500 ppm
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:58:29.432834 13159 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:58:29.433396 13159 webserver.cc:460] Webserver started at http://127.1.166.2:37587/ using document root <none> and password file <none>
I20260620 22:58:29.433624 13159 fs_manager.cc:359] Metadata directory not provided
I20260620 22:58:29.433674 13159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:58:29.433821 13159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:58:29.447842 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "30d35ac16cd8423aa221314436fbb461"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "6de746cd54697ad6f1b35155554a9924"
server_key_iv: "2b34cec39f29cc534e513142f092dee7"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.448336 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "30d35ac16cd8423aa221314436fbb461"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "6de746cd54697ad6f1b35155554a9924"
server_key_iv: "2b34cec39f29cc534e513142f092dee7"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.449739 13159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:58:29.462816 13176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.463080 13159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:58:29.463176 13159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "30d35ac16cd8423aa221314436fbb461"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "6de746cd54697ad6f1b35155554a9924"
server_key_iv: "2b34cec39f29cc534e513142f092dee7"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.463279 13159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:58:29.505952 13159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:58:29.508548 13159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:58:29.508777 13159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:58:29.509065 13159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:58:29.509533 13159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:58:29.509589 13159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.509618 13159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:58:29.509635 13159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.516963 13159 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:39883
I20260620 22:58:29.517618 13159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:58:29.520737 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 13159
I20260620 22:58:29.520854 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:58:29.521052 1688 external_mini_cluster.cc:1428] Setting key 47cd6ce77e4350fcdb997b7f7f60b30e
I20260620 22:58:29.519387 13289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:39883 every 8 connection(s)
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:58:29.530550 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.521448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:59551 (local address 127.1.166.62:36013)
0620 22:58:29.521491 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:29.521493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:29.521717 (+ 224us) server_negotiation.cc:407] Connection header received
0620 22:58:29.522951 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:29.522952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:29.522965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:29.522998 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:58:29.523804 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.524311 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.525424 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.525685 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.525742 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.527895 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:58:29.527897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:58:29.527897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:58:29.527906 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:58:29.529169 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:29.529563 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:29.529564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:29.529564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:29.529617 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:29.529848 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:29.529848 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:29.529849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:29.529971 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:58:29.530028 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:29.530359 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 22:58:29.530454 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:58:29.530786 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.521415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:36013 (local address 127.1.166.2:59551)
0620 22:58:29.521597 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:29.521631 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:58:29.522818 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:58:29.523197 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:58:29.523202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:58:29.523463 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:58:29.523709 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.523714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.524407 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.524409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.525255 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.525258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.525767 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.525768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.525907 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.526492 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:58:29.526502 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:58:29.527766 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:58:29.529249 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:58:29.529257 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:58:29.529260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:58:29.529494 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:58:29.529672 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:58:29.529673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:58:29.529674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:58:29.529792 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:58:29.530026 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:58:29.530033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:58:29.530202 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 22:58:29.530296 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 22:58:29.530492 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260620 22:58:29.531239 13290 heartbeater.cc:347] Connected to a master server at 127.1.166.62:36013
I20260620 22:58:29.531337 13290 heartbeater.cc:464] Registering TS with master...
I20260620 22:58:29.531522 13290 heartbeater.cc:511] Master 127.1.166.62:36013 requested a full tablet report, sending...
I20260620 22:58:29.531869 12941 ts_manager.cc:194] Registered new tserver with Master: 30d35ac16cd8423aa221314436fbb461 (127.1.166.2:39883)
I20260620 22:58:29.532238 12941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:59551
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:58:29.557669 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:57211/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:36013
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:58:29.660408 13296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:58:29.660758 13296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:58:29.660815 13296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:58:29.662503 13296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:58:29.662557 13296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:58:29.662598 13296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:58:29.664505 13296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.166.20:57211/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:36013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:58:29.664775 13296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:58:29.665006 13296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:58:29.672544 13302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:58:29.672786 13304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:58:29.672871 13301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:58:29.673879 13296 server_base.cc:1034] running on GCE node
I20260620 22:58:29.674258 13296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:58:29.674654 13296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:58:29.675856 13296 hybrid_clock.cc:648] HybridClock initialized: now 1781996309675835 us; error 39 us; skew 500 ppm
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:58:29.683960 13296 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:58:29.684597 13296 webserver.cc:460] Webserver started at http://127.1.166.3:44945/ using document root <none> and password file <none>
I20260620 22:58:29.684839 13296 fs_manager.cc:359] Metadata directory not provided
I20260620 22:58:29.684887 13296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:58:29.685050 13296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:58:29.697790 13296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "1e55c2eef25c457c4a064f3dd7af6500"
server_key_iv: "63bb2322ddb17ca349f3640b02136bc4"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.704164 13296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "1e55c2eef25c457c4a064f3dd7af6500"
server_key_iv: "63bb2322ddb17ca349f3640b02136bc4"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.706307 13296 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:58:29.734841 13311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.735071 13296 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 22:58:29.735179 13296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162"
format_stamp: "Formatted at 2026-06-20 22:58:29 on dist-test-slave-rdl9"
server_key: "1e55c2eef25c457c4a064f3dd7af6500"
server_key_iv: "63bb2322ddb17ca349f3640b02136bc4"
server_key_version: "kuduclusterkey@0"
I20260620 22:58:29.735303 13296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:58:29.757669 13296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:58:29.760175 13296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:58:29.760326 13296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:58:29.760536 13296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:58:29.760891 13296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:58:29.760928 13296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.760958 13296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:58:29.760979 13296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:58:29.767691 13296 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:45987
I20260620 22:58:29.768117 13296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:58:29.768163 13424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:45987 every 8 connection(s)
I20260620 22:58:29.775662 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 13296
I20260620 22:58:29.775801 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:58:29.776131 1688 external_mini_cluster.cc:1428] Setting key 347fe8c4d8766f56602c6517fd854f2a
I20260620 22:58:29.780049 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.768996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:56165 (local address 127.1.166.62:36013)
0620 22:58:29.769062 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:29.769064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:29.769342 (+ 278us) server_negotiation.cc:407] Connection header received
0620 22:58:29.770067 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:29.770070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:29.770093 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:29.770192 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:58:29.771100 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.771809 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.773269 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.773522 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.773609 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.776524 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:58:29.776527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:58:29.776527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:58:29.776537 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:58:29.778203 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:29.778763 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:29.778765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:29.778766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:29.778840 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:29.779162 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:29.779163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:29.779163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:29.779316 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:58:29.779388 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:29.779851 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 22:58:29.779953 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:58:29.780133 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.768906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:36013 (local address 127.1.166.3:56165)
0620 22:58:29.769165 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:29.769220 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 22:58:29.769855 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:58:29.770391 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:58:29.770398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:58:29.770667 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:58:29.770986 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.770992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.771912 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.771914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.773143 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.773146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.773638 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.773639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.773814 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.774444 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:58:29.774455 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:58:29.776414 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:58:29.778312 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:58:29.778324 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:58:29.778327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:58:29.778660 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:58:29.778936 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:58:29.778937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:58:29.778937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:58:29.779088 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:58:29.779460 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:58:29.779467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:58:29.779702 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 22:58:29.779798 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 22:58:29.779945 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
I20260620 22:58:29.780606 13425 heartbeater.cc:347] Connected to a master server at 127.1.166.62:36013
I20260620 22:58:29.780709 13425 heartbeater.cc:464] Registering TS with master...
I20260620 22:58:29.780916 13425 heartbeater.cc:511] Master 127.1.166.62:36013 requested a full tablet report, sending...
I20260620 22:58:29.781301 12941 ts_manager.cc:194] Registered new tserver with Master: ceff81a9ef8a47d49d4c9ecaa5bda162 (127.1.166.3:45987)
I20260620 22:58:29.781672 12941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:56165
I20260620 22:58:29.788290 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:58:29.797334 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.789736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.1.166.62:36013)
0620 22:58:29.789809 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:29.789811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:29.789887 (+ 76us) server_negotiation.cc:407] Connection header received
0620 22:58:29.789955 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:29.789956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:29.789965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:29.790013 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:58:29.790779 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.791265 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.792302 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.792492 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.792553 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.794558 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:58:29.794559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:58:29.794559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:58:29.794568 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:58:29.795921 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:29.796352 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:29.796353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:29.796353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:29.796423 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:58:29.796745 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:58:29.796746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:58:29.796747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:58:29.796881 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:58:29.797043 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:29.797192 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 22:58:29.797263 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:58:29.799299 12941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:58:29.801115 12941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:58:29.821118 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.814687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:39883 (local address 127.0.0.1:39488)
0620 22:58:29.816183 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:29.816236 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 22:58:29.816326 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:58:29.816776 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:58:29.816780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:58:29.816782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:58:29.816876 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.816879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.818146 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.818147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.819611 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.819614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.820763 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.820764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.820867 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.820883 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 22:58:29.820956 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:58:29.821017 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"thread_start_us":128,"threads_started":1}
I20260620 22:58:29.821535 13441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.815091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.1.166.2:39883)
0620 22:58:29.815295 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:29.815301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:29.816439 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 22:58:29.816495 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:29.816497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:29.816515 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:29.816587 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:58:29.816969 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.818036 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.819750 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.820559 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.820612 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.820697 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:29.820989 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 22:58:29.821445 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260620 22:58:29.822381 13224 tablet_service.cc:1467] Processing CreateTablet for tablet a40e81eee7964fbeb4511e24f68060f4 (DEFAULT_TABLE table=test-table [id=9d151588a06d46cb8f7e342a6c67ad8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:58:29.822782 13224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40e81eee7964fbeb4511e24f68060f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:58:29.825731 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.814691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:36771 (local address 127.0.0.1:50658)
0620 22:58:29.821331 (+ 6640us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:29.821342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:58:29.821377 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:58:29.821632 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:58:29.821633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:58:29.821634 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:58:29.821667 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.821871 (+ 204us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.823086 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.823087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.824193 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.824196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.825489 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.825490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.825552 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.825556 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:58:29.825598 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:58:29.825648 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6627,"thread_start_us":115,"threads_started":1}
I20260620 22:58:29.826040 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.814780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.1.166.1:36771)
0620 22:58:29.816728 (+ 1948us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:29.816734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:29.821430 (+ 4696us) server_negotiation.cc:407] Connection header received
0620 22:58:29.821472 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:29.821475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:29.821493 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:29.821562 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:58:29.821956 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.822989 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.824302 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.825125 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.825197 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.825254 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:29.825881 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 22:58:29.825934 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1910,"thread_start_us":54,"threads_started":1}
I20260620 22:58:29.826879 13087 tablet_service.cc:1467] Processing CreateTablet for tablet a40e81eee7964fbeb4511e24f68060f4 (DEFAULT_TABLE table=test-table [id=9d151588a06d46cb8f7e342a6c67ad8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:58:29.827248 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40e81eee7964fbeb4511e24f68060f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:58:29.831470 13444 tablet_bootstrap.cc:492] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf: Bootstrap starting.
I20260620 22:58:29.832294 13444 tablet_bootstrap.cc:654] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf: Neither blocks nor log segments found. Creating new log.
I20260620 22:58:29.832683 13444 log.cc:824] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf: Log is configured to *not* fsync() on all Append() calls
I20260620 22:58:29.833132 13443 tablet_bootstrap.cc:492] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Bootstrap starting.
I20260620 22:58:29.834180 13443 tablet_bootstrap.cc:654] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Neither blocks nor log segments found. Creating new log.
I20260620 22:58:29.834606 13443 log.cc:824] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Log is configured to *not* fsync() on all Append() calls
I20260620 22:58:29.835492 13444 tablet_bootstrap.cc:492] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf: No bootstrap required, opened a new log
I20260620 22:58:29.835614 13444 ts_tablet_manager.cc:1397] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 22:58:29.838588 13443 tablet_bootstrap.cc:492] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: No bootstrap required, opened a new log
I20260620 22:58:29.838675 13443 ts_tablet_manager.cc:1397] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 22:58:29.840101 13444 raft_consensus.cc:348] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:29.840440 13444 raft_consensus.cc:374] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:58:29.840533 13444 raft_consensus.cc:729] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de084f6d8b547c9b5516af47fdff6bf, State: Initialized, Role: FOLLOWER
I20260620 22:58:29.840899 13444 consensus_queue.cc:260] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:29.841388 13444 ts_tablet_manager.cc:1428] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260620 22:58:29.841565 13153 heartbeater.cc:503] Master 127.1.166.62:36013 was elected leader, sending a full tablet report...
I20260620 22:58:29.842880 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.814955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:45987 (local address 127.0.0.1:57810)
0620 22:58:29.825332 (+ 10377us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:29.825345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:58:29.825401 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:58:29.831189 (+ 5788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:58:29.831191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:58:29.831194 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:58:29.831257 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.831259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.837799 (+ 6540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.837801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.839020 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:29.839025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.842589 (+ 3564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.842590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:29.842675 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.842679 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:58:29.842732 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 22:58:29.842802 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10351,"thread_start_us":59,"threads_started":1}
I20260620 22:58:29.842926 13443 raft_consensus.cc:348] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:29.843118 13443 raft_consensus.cc:374] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:58:29.843148 13443 raft_consensus.cc:729] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d35ac16cd8423aa221314436fbb461, State: Initialized, Role: FOLLOWER
I20260620 22:58:29.843274 13443 consensus_queue.cc:260] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:29.843562 13443 ts_tablet_manager.cc:1428] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 22:58:29.843688 13290 heartbeater.cc:503] Master 127.1.166.62:36013 was elected leader, sending a full tablet report...
I20260620 22:58:29.843881 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:29.815279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.1.166.3:45987)
0620 22:58:29.815428 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:29.815431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:29.827366 (+ 11935us) server_negotiation.cc:407] Connection header received
0620 22:58:29.827417 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:29.827420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:29.827438 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:29.827522 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:58:29.836506 (+ 8984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.837660 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.841082 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:29.841938 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:29.842019 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:29.842102 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:29.843737 (+ 1635us) server_negotiation.cc:299] Negotiation successful
0620 22:58:29.843800 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260620 22:58:29.844714 13359 tablet_service.cc:1467] Processing CreateTablet for tablet a40e81eee7964fbeb4511e24f68060f4 (DEFAULT_TABLE table=test-table [id=9d151588a06d46cb8f7e342a6c67ad8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:58:29.845113 13359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40e81eee7964fbeb4511e24f68060f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:58:29.853411 13449 tablet_bootstrap.cc:492] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Bootstrap starting.
I20260620 22:58:29.854951 13449 tablet_bootstrap.cc:654] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Neither blocks nor log segments found. Creating new log.
I20260620 22:58:29.855396 13449 log.cc:824] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Log is configured to *not* fsync() on all Append() calls
I20260620 22:58:29.856830 13449 tablet_bootstrap.cc:492] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: No bootstrap required, opened a new log
I20260620 22:58:29.856974 13449 ts_tablet_manager.cc:1397] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 22:58:29.859781 13449 raft_consensus.cc:348] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:29.860034 13449 raft_consensus.cc:374] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:58:29.860073 13449 raft_consensus.cc:729] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceff81a9ef8a47d49d4c9ecaa5bda162, State: Initialized, Role: FOLLOWER
I20260620 22:58:29.860249 13449 consensus_queue.cc:260] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:29.860625 13449 ts_tablet_manager.cc:1428] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 22:58:29.860673 13425 heartbeater.cc:503] Master 127.1.166.62:36013 was elected leader, sending a full tablet report...
Jun 20 22:58:29 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
W20260620 22:58:30.009296 13154 tablet.cc:2369] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:58:30.018725 13426 tablet.cc:2369] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:58:30.020505 13291 tablet.cc:2369] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:58:30.189668 13448 raft_consensus.cc:482] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:58:30.189925 13448 raft_consensus.cc:504] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:30.190450 13448 leader_election.cc:288] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1de084f6d8b547c9b5516af47fdff6bf (127.1.166.1:36771), ceff81a9ef8a47d49d4c9ecaa5bda162 (127.1.166.3:45987)
I20260620 22:58:30.194219 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:30.190727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:52189 (local address 127.1.166.1:36771)
0620 22:58:30.190773 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:30.190775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:30.190949 (+ 174us) server_negotiation.cc:407] Connection header received
0620 22:58:30.191045 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:30.191046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:30.191056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:30.191083 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:58:30.191283 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.191965 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.193323 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.193843 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.193886 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:30.193897 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:30.194099 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 22:58:30.194145 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:58:30.194430 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:30.190945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:52537 (local address 127.1.166.3:45987)
0620 22:58:30.190987 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:30.190989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:30.191126 (+ 137us) server_negotiation.cc:407] Connection header received
0620 22:58:30.191221 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:30.191222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:30.191234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:30.191260 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:58:30.191460 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.192168 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.193472 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.194004 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.194052 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:30.194061 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:30.194289 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 22:58:30.194330 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:58:30.194682 13453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:30.190807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:45987 (local address 127.1.166.2:52537)
0620 22:58:30.191063 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:30.191073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:58:30.191152 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:58:30.191320 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:58:30.191321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:58:30.191323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:58:30.191394 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:30.191396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.192287 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.192288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:30.193366 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:30.193370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.194122 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.194123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:30.194200 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:30.194203 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:58:30.194550 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 22:58:30.194598 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":48,"threads_started":1}
I20260620 22:58:30.194783 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:30.190637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:36771 (local address 127.1.166.2:52189)
0620 22:58:30.190870 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:30.190881 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:58:30.190975 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:58:30.191145 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:58:30.191146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:58:30.191147 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:58:30.191207 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:30.191209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.192065 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.192066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:30.193190 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:58:30.193192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.193952 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.193952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:58:30.194022 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:30.194026 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:58:30.194648 (+ 622us) client_negotiation.cc:240] Negotiation successful
0620 22:58:30.194694 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":5,"thread_start_us":72,"threads_started":1}
I20260620 22:58:30.195219 13379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40e81eee7964fbeb4511e24f68060f4" candidate_uuid: "30d35ac16cd8423aa221314436fbb461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" is_pre_election: true
I20260620 22:58:30.195410 13379 raft_consensus.cc:2454] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d35ac16cd8423aa221314436fbb461 in term 0.
I20260620 22:58:30.195439 13106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40e81eee7964fbeb4511e24f68060f4" candidate_uuid: "30d35ac16cd8423aa221314436fbb461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1de084f6d8b547c9b5516af47fdff6bf" is_pre_election: true
I20260620 22:58:30.195586 13106 raft_consensus.cc:2454] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d35ac16cd8423aa221314436fbb461 in term 0.
I20260620 22:58:30.195806 13178 leader_election.cc:302] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d35ac16cd8423aa221314436fbb461, ceff81a9ef8a47d49d4c9ecaa5bda162; no voters:
I20260620 22:58:30.195991 13448 raft_consensus.cc:2781] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:58:30.196038 13448 raft_consensus.cc:482] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:58:30.196058 13448 raft_consensus.cc:3037] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:58:30.196943 13448 raft_consensus.cc:504] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:30.197062 13448 leader_election.cc:288] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [CANDIDATE]: Term 1 election: Requested vote from peers 1de084f6d8b547c9b5516af47fdff6bf (127.1.166.1:36771), ceff81a9ef8a47d49d4c9ecaa5bda162 (127.1.166.3:45987)
I20260620 22:58:30.197296 13106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40e81eee7964fbeb4511e24f68060f4" candidate_uuid: "30d35ac16cd8423aa221314436fbb461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1de084f6d8b547c9b5516af47fdff6bf"
I20260620 22:58:30.197359 13106 raft_consensus.cc:3037] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:58:30.198159 13106 raft_consensus.cc:2454] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d35ac16cd8423aa221314436fbb461 in term 1.
I20260620 22:58:30.198324 13379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40e81eee7964fbeb4511e24f68060f4" candidate_uuid: "30d35ac16cd8423aa221314436fbb461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162"
I20260620 22:58:30.198499 13178 leader_election.cc:302] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1de084f6d8b547c9b5516af47fdff6bf, 30d35ac16cd8423aa221314436fbb461; no voters:
I20260620 22:58:30.198532 13379 raft_consensus.cc:3037] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:58:30.199819 13379 raft_consensus.cc:2454] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d35ac16cd8423aa221314436fbb461 in term 1.
I20260620 22:58:30.200115 13448 raft_consensus.cc:2781] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:58:30.200279 13448 raft_consensus.cc:686] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 1 LEADER]: Becoming Leader. State: Replica: 30d35ac16cd8423aa221314436fbb461, State: Running, Role: LEADER
I20260620 22:58:30.200382 13448 consensus_queue.cc:237] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } }
I20260620 22:58:30.201148 12941 catalog_manager.cc:5495] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d35ac16cd8423aa221314436fbb461 (127.1.166.2). New cstate: current_term: 1 leader_uuid: "30d35ac16cd8423aa221314436fbb461" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d35ac16cd8423aa221314436fbb461" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 } health_report { overall_health: UNKNOWN } } }
I20260620 22:58:30.264885 13441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:30.260273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.1.166.2:39883)
0620 22:58:30.260323 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:58:30.260325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:58:30.260637 (+ 312us) server_negotiation.cc:407] Connection header received
0620 22:58:30.260705 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:58:30.260706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:58:30.260731 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:58:30.260826 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:58:30.260963 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.261664 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.262812 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:58:30.263128 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:58:30.263214 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:58:30.264244 (+ 1030us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:58:30.264404 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:58:30.264498 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 22:58:30.264773 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 22:58:30.264798 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 22:58:30 dist-test-slave-rdl9 krb5kdc[12464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:58:30.266963 13379 raft_consensus.cc:1261] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 1 FOLLOWER]: Refusing update from remote peer 30d35ac16cd8423aa221314436fbb461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:58:30.267009 13106 raft_consensus.cc:1261] T a40e81eee7964fbeb4511e24f68060f4 P 1de084f6d8b547c9b5516af47fdff6bf [term 1 FOLLOWER]: Refusing update from remote peer 30d35ac16cd8423aa221314436fbb461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:58:30.267315 13448 consensus_queue.cc:1035] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1de084f6d8b547c9b5516af47fdff6bf" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:58:30.269351 13448 consensus_queue.cc:1035] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceff81a9ef8a47d49d4c9ecaa5bda162" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:58:30.274632 13458 mvcc.cc:205] Tried to move back new op lower bound from 7299056886850568192 to 7299056886581370880. Current Snapshot: MvccSnapshot[applied={T|T < 7299056886850568192}]
I20260620 22:58:30.275686 13461 mvcc.cc:205] Tried to move back new op lower bound from 7299056886850568192 to 7299056886581370880. Current Snapshot: MvccSnapshot[applied={T|T < 7299056886850568192}]
I20260620 22:58:30.275893 13459 mvcc.cc:205] Tried to move back new op lower bound from 7299056886850568192 to 7299056886581370880. Current Snapshot: MvccSnapshot[applied={T|T < 7299056886850568192}]
I20260620 22:58:30.282110 12941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:58:30.282228 12941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:58:30.283206 12941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5476aa512ed3467c8939c5888d276839: Sending DeleteTablet for 3 replicas of tablet a40e81eee7964fbeb4511e24f68060f4
I20260620 22:58:30.283608 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet a40e81eee7964fbeb4511e24f68060f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:58:30 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:50658
I20260620 22:58:30.284164 13359 tablet_service.cc:1514] Processing DeleteTablet for tablet a40e81eee7964fbeb4511e24f68060f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:58:30 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:57810
I20260620 22:58:30.284366 13224 tablet_service.cc:1514] Processing DeleteTablet for tablet a40e81eee7964fbeb4511e24f68060f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:58:30 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:39488
I20260620 22:58:30.284607 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 13022
I20260620 22:58:30.291970 13470 tablet_replica.cc:330] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: stopping tablet replica
I20260620 22:58:30.292223 13470 raft_consensus.cc:2229] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:58:30.292512 13470 raft_consensus.cc:2258] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:58:30.292835 13469 tablet_replica.cc:330] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: stopping tablet replica
I20260620 22:58:30.293013 13469 raft_consensus.cc:2229] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:58:30.293183 13469 raft_consensus.cc:2258] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:58:30.293241 13470 ts_tablet_manager.cc:1905] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:58:30.293676 13469 ts_tablet_manager.cc:1905] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:58:30.295058 13470 ts_tablet_manager.cc:1918] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:58:30.295250 13470 log.cc:1196] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/a40e81eee7964fbeb4511e24f68060f4
I20260620 22:58:30.295339 13469 ts_tablet_manager.cc:1918] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:58:30.295471 13469 log.cc:1196] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/a40e81eee7964fbeb4511e24f68060f4
I20260620 22:58:30.295593 13470 ts_tablet_manager.cc:1939] T a40e81eee7964fbeb4511e24f68060f4 P 30d35ac16cd8423aa221314436fbb461: Deleting consensus metadata
I20260620 22:58:30.295751 13469 ts_tablet_manager.cc:1939] T a40e81eee7964fbeb4511e24f68060f4 P ceff81a9ef8a47d49d4c9ecaa5bda162: Deleting consensus metadata
W20260620 22:58:30.295830 12926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:36771 (error 108)
I20260620 22:58:30.296203 12926 catalog_manager.cc:4841] TS ceff81a9ef8a47d49d4c9ecaa5bda162 (127.1.166.3:45987): tablet a40e81eee7964fbeb4511e24f68060f4 (table test-table [id=9d151588a06d46cb8f7e342a6c67ad8a]) successfully deleted
I20260620 22:58:30.296897 12927 catalog_manager.cc:4841] TS 30d35ac16cd8423aa221314436fbb461 (127.1.166.2:39883): tablet a40e81eee7964fbeb4511e24f68060f4 (table test-table [id=9d151588a06d46cb8f7e342a6c67ad8a]) successfully deleted
I20260620 22:58:30.297124 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 13159
I20260620 22:58:30.301855 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:58:30.301702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:36771 (local address 127.0.0.1:50674)
0620 22:58:30.301758 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 22:58:30.301811 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:36771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 22:58:30.302004 12926 catalog_manager.cc:4584] TS 1de084f6d8b547c9b5516af47fdff6bf (127.1.166.1:36771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a40e81eee7964fbeb4511e24f68060f4: Network error: Client connection negotiation failed: client connection to 127.1.166.1:36771: connect: Connection refused (error 111)
I20260620 22:58:30.305131 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 13296
I20260620 22:58:30.310467 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 12909
I20260620 22:58:30.316875 1688 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 22:58:30.652343 1688 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 22:58:30.652457 1688 mini_ranger.cc:66] Stopping Ranger...
I20260620 22:58:30.988461 1688 mini_ranger.cc:68] Stopped Ranger
2026-06-20 22:58:30.988 UTC [12494] LOG: received smart shutdown request
2026-06-20 22:58:30.991 UTC [12494] LOG: background worker "logical replication launcher" (PID 12503) exited with exit code 1
2026-06-20 22:58:30.991 UTC [12498] LOG: shutting down
2026-06-20 22:58:31.149 UTC [12494] LOG: database system is shut down
2026-06-20T22:58:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53080 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:58:31 dist-test-slave-rdl9 krb5kdc[13481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:58:31 dist-test-slave-rdl9 krb5kdc[13481](info): set up 2 sockets
Jun 20 22:58:31 dist-test-slave-rdl9 krb5kdc[13481](info): commencing operation
krb5kdc: starting...
W20260620 22:58:32.226234 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:58:32 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:58:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/rangeradmin_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/rangeradmin_127.1.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/rangerlookup_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/rangerlookup_127.1.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/HTTP_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/HTTP_127.1.166.20@KRBTEST.COM.keytab.
I20260620 22:58:32.392932 1688 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskE9BVnS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/postgres -l logfile start
2026-06-20 22:58:34.926 UTC [13511] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 22:58:34.926 UTC [13511] LOG: listening on IPv4 address "127.1.166.20", port 36029
2026-06-20 22:58:34.929 UTC [13511] LOG: listening on Unix socket "/tmp/.s.PGSQL.36029"
2026-06-20 22:58:34.950 UTC [13514] LOG: database system was shut down at 2026-06-20 22:58:33 UTC
2026-06-20 22:58:34.954 UTC [13511] LOG: database system is ready to accept connections
I20260620 22:58:35.858412 1688 mini_postgres.cc:96] Postgres bound to 36029
2026-06-20 22:58:35.864 UTC [13522] FATAL: database "slave" does not exist
127.1.166.20:36029 - accepting connections
I20260620 22:58:35.864919 1688 mini_ranger.cc:161] Starting Ranger...
I20260620 22:58:35.881290 1688 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 22:58:36.369469 1688 mini_ranger.cc:87] Created ranger Postgres database
I20260620 22:58:36.369567 1688 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-admin
2026-06-20 22:58:36,646 [I] DB FLAVOR :POSTGRES
2026-06-20 22:58:36,647 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:58:36,647 [I] Checking connection..
2026-06-20 22:58:36,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 22:58:36,922 [I] Checking connection passed.
2026-06-20 22:58:36,922 [I] --------- Verifying version history table ---------
2026-06-20 22:58:36,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:58:37,212 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:58:37,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:58:37,475 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:58:37,475 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 22:58:37,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 22:58:37.753 UTC [13586] WARNING: there is no transaction in progress
2026-06-20 22:58:37,760 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 22:58:37,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:58:38,048 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 22:58:38,048 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:58:38,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 22:58:38,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 22:58:38,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rdl9.c.gcp-upstream.internal','N') ;"
2026-06-20 22:58:38,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:58:39,142 [I] Table x_portal_user does not exist in database ranger
2026-06-20 22:58:39,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:58:39,417 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 22:58:39,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:58:39,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:58:39,983 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 22:58:39,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 22:58:41.085 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.104 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.122 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.138 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.154 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.175 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.242 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.252 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.269 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.280 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.293 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.310 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.321 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.329 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.337 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:41.344 UTC [13694] WARNING: there is no transaction in progress
2026-06-20 22:58:42,104 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 22:58:42,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 22:58:42,383 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 22:58:42,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:58:42,672 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 22:58:42,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:58:42,945 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 22:58:42,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:58:43,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:58:43,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rdl9.c.gcp-upstream.internal';"
2026-06-20 22:58:43,817 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 22:58:43,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 22:58:44,070 [I] DB_PATCHES have already been applied
I20260620 22:58:44.077730 1688 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-admin:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskE9BVnS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 22:58:44.077807 1688 mini_ranger.cc:193] Using host: 127.1.166.20
I20260620 22:58:44.081028 1688 mini_ranger.cc:227] Ranger admin URL: http://127.1.166.20:51811
Jun 20, 2026 10:58:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:58:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:58:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 10:58:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 10:58:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 10:58:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 10:58:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 22:58:46.124680 1688 mini_ranger.cc:160] Time spent starting Ranger: real 10.260s user 0.006s sys 0.007s
Jun 20, 2026 10:59:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 10:59:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 10:59:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 10:59:08 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 10:59:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 22:59:10.589955 1688 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/rangerkms_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/rangerkms_127.1.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.166.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.166.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.166.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/HTTP_127.1.166.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.166.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/HTTP_127.1.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 22:59:10 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 22:59:10.671577 1688 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 22:59:10.682196 1688 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 22:59:11.497100 1688 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 22:59:11.497181 1688 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms
I20260620 22:59:11.497193 1688 mini_ranger_kms.cc:227] Using postgres at 127.1.166.20:36029
2026-06-20 22:59:11,660 [I] DB FLAVOR :POSTGRES
2026-06-20 22:59:11,661 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:59:11,661 [I] Checking connection
2026-06-20 22:59:11,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 22:59:11,941 [I] connection success
2026-06-20 22:59:11,941 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 22:59:11,941 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 22:59:11,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 22:59:12,229 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 22:59:12,230 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:59:12,230 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 22:59:12,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.166.20:36029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 22:59:12,534 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 22:59:13.093369 1688 mini_ranger_kms.cc:310] Created kms service
I20260620 22:59:13.343276 1688 mini_ranger_kms.cc:326] Created kudu user
I20260620 22:59:13.400221 1688 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 22:59:13 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:13.870992 1688 mini_ranger_kms.cc:384] Added ranger policy
I20260620 22:59:13.871224 1688 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms:/tmp/dist-test-taskE9BVnS/build/release/bin/postgresql.jar:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskE9BVnS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 22:59:13.871243 1688 mini_ranger_kms.cc:240] Using host: 127.1.166.20
I20260620 22:59:13.874553 1688 mini_ranger_kms.cc:276] Ranger KMS PID: 13876
I20260620 22:59:13.874639 1688 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.166.20:52007
Jun 20, 2026 10:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 10:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 10:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 10:59:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 22:59:16.042070 1688 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.370s user 0.006s sys 0.135s
I20260620 22:59:16.042168 1688 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 22:59:16.042214 1688 mini_ranger_kms.cc:401] 127.1.166.20:52007/kms/v1/keys
Jun 20 22:59:16 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.166.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 22:59:16 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.166.20@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
Jun 20 22:59:17 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.166.20@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
Jun 20 22:59:19 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:59:19.837540 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35827
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:35827
--ranger_config_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:52007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:19.951249 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:19.951503 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:19.951527 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:19.952819 13926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:19.952890 13926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:19.952906 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:19.952922 13926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:19.952935 13926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:19.954818 13926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.166.20:52007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:35827
--ranger_config_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:19.955116 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:19.955372 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:19.960985 13932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:19.961071 13934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:19.961127 13926 server_base.cc:1034] running on GCE node
W20260620 22:59:19.961175 13931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:19.961680 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:19.961980 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:19.963146 13926 hybrid_clock.cc:648] HybridClock initialized: now 1781996359963129 us; error 36 us; skew 500 ppm
Jun 20 22:59:19 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:19.965962 13926 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:59:19.966503 13926 webserver.cc:460] Webserver started at http://127.1.166.62:40015/ using document root <none> and password file <none>
I20260620 22:59:19.966717 13926 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:19.966758 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:19.966885 13926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:59:19 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:20.033303 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "b5284d3748404d8aa1c41c39e31a9854"
format_stamp: "Formatted at 2026-06-20 22:59:20 on dist-test-slave-rdl9"
server_key: "429b8aac261fe0f8a86b729a178c5ebd"
server_key_iv: "be21de39021fa9015faaaa2260f6f74e"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:20.033694 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "b5284d3748404d8aa1c41c39e31a9854"
format_stamp: "Formatted at 2026-06-20 22:59:20 on dist-test-slave-rdl9"
server_key: "429b8aac261fe0f8a86b729a178c5ebd"
server_key_iv: "be21de39021fa9015faaaa2260f6f74e"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:20.035800 13926 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 22:59:20.055430 13941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:20.055691 13926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:59:20.055791 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "b5284d3748404d8aa1c41c39e31a9854"
format_stamp: "Formatted at 2026-06-20 22:59:20 on dist-test-slave-rdl9"
server_key: "429b8aac261fe0f8a86b729a178c5ebd"
server_key_iv: "be21de39021fa9015faaaa2260f6f74e"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:20.055919 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:20.074607 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:20.076773 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:20.077147 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:20.083066 13926 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:35827
I20260620 22:59:20.083184 13993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:35827 every 8 connection(s)
I20260620 22:59:20.083557 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:59:20.084297 13994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:20.086189 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 13926
I20260620 22:59:20.086406 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:59:20.086651 1688 external_mini_cluster.cc:1428] Setting key 68b1a0860c35cad2824158b03da67497
I20260620 22:59:20.087385 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Bootstrap starting.
I20260620 22:59:20.088335 13994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:20.088704 13994 log.cc:824] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:20.089612 13994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: No bootstrap required, opened a new log
I20260620 22:59:20.092532 13994 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } }
I20260620 22:59:20.092774 13994 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:20.092813 13994 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5284d3748404d8aa1c41c39e31a9854, State: Initialized, Role: FOLLOWER
I20260620 22:59:20.092998 13994 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } }
I20260620 22:59:20.093215 13994 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:59:20.093515 13994 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:59:20.093631 13994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:20.094713 13994 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } }
I20260620 22:59:20.094858 13994 leader_election.cc:302] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5284d3748404d8aa1c41c39e31a9854; no voters:
I20260620 22:59:20.095057 13994 leader_election.cc:288] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:59:20.095203 13999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:20.095407 13999 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 1 LEADER]: Becoming Leader. State: Replica: b5284d3748404d8aa1c41c39e31a9854, State: Running, Role: LEADER
I20260620 22:59:20.095544 13999 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } }
Jun 20 22:59:20 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:20.095942 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5284d3748404d8aa1c41c39e31a9854. Latest consensus state: current_term: 1 leader_uuid: "b5284d3748404d8aa1c41c39e31a9854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } } }
I20260620 22:59:20.096010 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:20.096467 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5284d3748404d8aa1c41c39e31a9854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } } }
I20260620 22:59:20.096563 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:20.096890 13994 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:20.097358 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:20.098132 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:20.098868 13994 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:59:20.104226 14002 catalog_manager.cc:1347] Generated new cluster ID: 2becfb6924f24763bbefcb55df50b519
I20260620 22:59:20.104249 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:20.087450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.1.166.62:35827)
0620 22:59:20.087713 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:20.087724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:20.087770 (+ 46us) server_negotiation.cc:407] Connection header received
0620 22:59:20.088432 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:20.088442 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:20.088524 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:20.088710 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:20.089886 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:20.091329 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:20.092537 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:20.093171 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:20.093287 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:20.096706 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:20.096711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:20.096712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:20.096726 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:20.098783 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:20.099243 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:20.099248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:20.099248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:20.099351 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:20.099572 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:20.099573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:20.099573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:20.101579 (+ 2006us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:20.103169 (+ 1590us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:20.103328 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 22:59:20.103754 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":134,"threads_started":1}
I20260620 22:59:20.104396 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:59:20.112707 14002 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:59:20.113476 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:59:20.122526 14002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Generated new TSK 0
I20260620 22:59:20.122857 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:59:21.001212 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:52007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35827
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:21.091749 14038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:21.092029 14038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:21.092057 14038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:21.093425 14038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:21.093490 14038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:21.093520 14038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:59:21.095294 14038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.166.20:52007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:21.095537 14038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:21.095817 14038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:21.102567 14043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:21.102573 14044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:21.102825 14046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:21.103097 14038 server_base.cc:1034] running on GCE node
I20260620 22:59:21.103331 14038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:21.103579 14038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:21.104743 14038 hybrid_clock.cc:648] HybridClock initialized: now 1781996361104712 us; error 44 us; skew 500 ppm
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:21.107707 14038 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:59:21.108390 14038 webserver.cc:460] Webserver started at http://127.1.166.1:35867/ using document root <none> and password file <none>
I20260620 22:59:21.108690 14038 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:21.108781 14038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:21.108942 14038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:21.128638 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "c3623beec5644a4e8820449d83d8f0cd"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "3002b2b99b4211fe5b0217773e6e50cc"
server_key_iv: "bab2609735a5be0fbbce71cf433bac6c"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.129245 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "c3623beec5644a4e8820449d83d8f0cd"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "3002b2b99b4211fe5b0217773e6e50cc"
server_key_iv: "bab2609735a5be0fbbce71cf433bac6c"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.134845 14038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260620 22:59:21.150521 14053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.150729 14038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 22:59:21.150820 14038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "c3623beec5644a4e8820449d83d8f0cd"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "3002b2b99b4211fe5b0217773e6e50cc"
server_key_iv: "bab2609735a5be0fbbce71cf433bac6c"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.150924 14038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:21.164528 14038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:21.166611 14038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:21.166760 14038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:21.166982 14038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:21.167335 14038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:21.167373 14038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.167403 14038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:21.167454 14038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.173774 14038 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:44809
I20260620 22:59:21.174340 14038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:59:21.175690 14167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:44809 every 8 connection(s)
I20260620 22:59:21.177861 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 14038
I20260620 22:59:21.177981 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:59:21.178157 1688 external_mini_cluster.cc:1428] Setting key 1a289893b1683bd471283d5d14447ae6
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:21.191121 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:21.176631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:56601 (local address 127.1.166.62:35827)
0620 22:59:21.176749 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:21.176751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:21.176943 (+ 192us) server_negotiation.cc:407] Connection header received
0620 22:59:21.177560 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:21.177561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:21.177577 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:21.177620 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:21.180777 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.181445 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.182731 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.183044 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.183156 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:21.185520 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:21.185522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:21.185523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:21.185532 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:21.187159 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:21.189623 (+ 2464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:21.189625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:21.189625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:21.189718 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:21.190074 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:21.190076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:21.190076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:21.190257 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:21.190310 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:21.190858 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 22:59:21.191003 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260620 22:59:21.191731 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:21.176629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35827 (local address 127.1.166.1:56601)
0620 22:59:21.176810 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:21.176854 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:21.177432 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:21.178668 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:21.178673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:21.180301 (+ 1628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:21.180639 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:21.180649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.181600 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.181601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:21.182629 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:21.182634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.183196 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.183197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:21.183337 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:21.184000 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:21.184011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:21.185359 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:21.188998 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:21.189016 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:21.189020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:21.189469 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:21.189804 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:21.189805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:21.189805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:21.189989 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:21.190408 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:21.190417 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:21.190666 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 22:59:21.191244 (+ 578us) client_negotiation.cc:240] Negotiation successful
0620 22:59:21.191470 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260620 22:59:21.192384 14168 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35827
I20260620 22:59:21.192490 14168 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:21.192687 14168 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:21.193253 13958 ts_manager.cc:194] Registered new tserver with Master: c3623beec5644a4e8820449d83d8f0cd (127.1.166.1:44809)
I20260620 22:59:21.194200 13958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:56601
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:59:21.214078 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:52007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35827
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:21.336302 14175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:21.336709 14175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:21.336812 14175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:21.338797 14175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:21.338938 14175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:21.339023 14175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:59:21.341476 14175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.166.20:52007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:21.342029 14175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:21.342389 14175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:21.355612 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:21.355865 14175 server_base.cc:1034] running on GCE node
W20260620 22:59:21.355901 14182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:21.356281 14183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:21.356639 14175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:21.356990 14175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:21.358181 14175 hybrid_clock.cc:648] HybridClock initialized: now 1781996361358154 us; error 42 us; skew 500 ppm
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:21.361470 14175 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:59:21.362046 14175 webserver.cc:460] Webserver started at http://127.1.166.2:39289/ using document root <none> and password file <none>
I20260620 22:59:21.362540 14175 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:21.362664 14175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:21.362854 14175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:21.381765 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "315832584f5c4e08be508ed19cbf033d"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "a8276e4e830c15a85427faa6d09487d3"
server_key_iv: "e5f4b90dad47e4f219de06a73b4e8e05"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.383850 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "315832584f5c4e08be508ed19cbf033d"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "a8276e4e830c15a85427faa6d09487d3"
server_key_iv: "e5f4b90dad47e4f219de06a73b4e8e05"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.385880 14175 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 22:59:21.402586 14192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.402863 14175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:21.403213 14175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "315832584f5c4e08be508ed19cbf033d"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "a8276e4e830c15a85427faa6d09487d3"
server_key_iv: "e5f4b90dad47e4f219de06a73b4e8e05"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.403373 14175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:21.439939 14175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:21.442574 14175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:21.442817 14175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:21.443122 14175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:21.443519 14175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:21.443621 14175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.443701 14175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:21.443758 14175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.452600 14175 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:46289
I20260620 22:59:21.453332 14175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:59:21.456499 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:46289 every 8 connection(s)
I20260620 22:59:21.463111 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 14175
I20260620 22:59:21.463225 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:59:21.463406 1688 external_mini_cluster.cc:1428] Setting key 820d4464a9263f827e0dd08cfabeadf9
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:21.469439 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:21.457546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:55945 (local address 127.1.166.62:35827)
0620 22:59:21.457592 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:21.457594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:21.457784 (+ 190us) server_negotiation.cc:407] Connection header received
0620 22:59:21.458490 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:21.458491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:21.458502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:21.458531 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:21.459426 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.460249 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.461725 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.462041 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.462080 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:21.465623 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:21.465625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:21.465625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:21.465634 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:21.467370 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:21.467954 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:21.467955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:21.467955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:21.468021 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:21.468601 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:21.468602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:21.468602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:21.468758 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:21.468803 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:21.469216 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 22:59:21.469321 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:59:21.470431 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:21.457266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35827 (local address 127.1.166.2:55945)
0620 22:59:21.457671 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:21.457712 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:21.458381 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:21.458739 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:21.458743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:21.459046 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:21.459315 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:21.459320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.460388 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.460389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:21.461628 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:21.461630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.462182 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.462183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:21.462326 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:21.462922 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:21.462933 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:21.465492 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:21.467502 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:21.467514 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:21.467518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:21.467865 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:21.468085 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:21.468086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:21.468086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:21.468211 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:21.468848 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:21.468856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:21.469065 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 22:59:21.469827 (+ 762us) client_negotiation.cc:240] Negotiation successful
0620 22:59:21.470016 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":91,"threads_started":1}
I20260620 22:59:21.471164 14306 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35827
I20260620 22:59:21.471263 14306 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:21.471477 14306 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:21.471877 13958 ts_manager.cc:194] Registered new tserver with Master: 315832584f5c4e08be508ed19cbf033d (127.1.166.2:46289)
I20260620 22:59:21.472326 13958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:55945
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:59:21.501395 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:52007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35827
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:21.648819 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:21.649200 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:21.649287 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:21.651772 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:21.651932 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:21.652024 14312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:59:21.654829 14312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.166.20:52007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:21.655350 14312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:21.655685 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:21.663131 14318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:21.663596 14320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:21.663132 14317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:21.665169 14312 server_base.cc:1034] running on GCE node
I20260620 22:59:21.665540 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:21.665875 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:21.667042 14312 hybrid_clock.cc:648] HybridClock initialized: now 1781996361667021 us; error 36 us; skew 500 ppm
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:21.670555 14312 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:59:21.671128 14312 webserver.cc:460] Webserver started at http://127.1.166.3:38601/ using document root <none> and password file <none>
I20260620 22:59:21.671376 14312 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:21.671437 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:21.671566 14312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:21.693986 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "99db9b7da9cf409480a4d801e55d41e5"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "217937721a8554c08558b972121955f6"
server_key_iv: "5fd63d2e05e79656f939b71010385a94"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.694578 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "99db9b7da9cf409480a4d801e55d41e5"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "217937721a8554c08558b972121955f6"
server_key_iv: "5fd63d2e05e79656f939b71010385a94"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.697243 14312 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 22:59:21.714504 14327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.714690 14312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:21.714787 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "99db9b7da9cf409480a4d801e55d41e5"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "217937721a8554c08558b972121955f6"
server_key_iv: "5fd63d2e05e79656f939b71010385a94"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.714891 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:21.733954 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:21.736811 14312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:21.737053 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:21.737324 14312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:21.737726 14312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:21.737809 14312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.737879 14312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:21.737933 14312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.744266 14312 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:37873
I20260620 22:59:21.745010 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:59:21.745738 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 14312
I20260620 22:59:21.745849 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:59:21.746054 1688 external_mini_cluster.cc:1428] Setting key 0b531d5830af7eeaaf72935838337fdc
I20260620 22:59:21.754769 14440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:37873 every 8 connection(s)
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:21.765726 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:21.756141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:60281 (local address 127.1.166.62:35827)
0620 22:59:21.756189 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:21.756191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:21.756472 (+ 281us) server_negotiation.cc:407] Connection header received
0620 22:59:21.757012 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:21.757014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:21.757030 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:21.757129 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:21.757858 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.758701 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.760207 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.760421 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.760476 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:21.762878 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:21.762880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:21.762880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:21.762888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:21.764274 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:21.764717 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:21.764718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:21.764719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:21.764778 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:21.765018 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:21.765019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:21.765020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:21.765142 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:21.765207 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:21.765522 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 22:59:21.765614 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:59:21.765893 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:21.756135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35827 (local address 127.1.166.3:60281)
0620 22:59:21.756326 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:21.756369 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:21.756895 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:21.757212 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:21.757228 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:21.757472 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:21.757750 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:21.757755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.758821 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.758822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:21.760112 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:21.760115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:21.760515 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:21.760516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:21.760651 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:21.761193 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:21.761204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:21.762751 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:21.764378 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:21.764388 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:21.764391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:21.764638 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:21.764840 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:21.764841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:21.764841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:21.764959 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:21.765210 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:21.765218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:21.765390 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 22:59:21.765494 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 22:59:21.765660 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260620 22:59:21.766443 14441 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35827
I20260620 22:59:21.766564 14441 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:21.766760 14441 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:21.767189 13958 ts_manager.cc:194] Registered new tserver with Master: 99db9b7da9cf409480a4d801e55d41e5 (127.1.166.3:37873)
I20260620 22:59:21.767652 13958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:60281
I20260620 22:59:21.769469 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:59:21.770186 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 14038
I20260620 22:59:21.776476 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 14175
I20260620 22:59:21.780623 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 14312
I20260620 22:59:21.784734 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 13926
I20260620 22:59:21.790088 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35827
--webserver_interface=127.1.166.62
--webserver_port=40015
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:35827
--ranger_config_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:52007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.166.20:52007/kms with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:21.878770 14444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:21.879024 14444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:21.879060 14444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:21.880412 14444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:21.880481 14444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:21.880499 14444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:21.880515 14444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:21.880532 14444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:21.882447 14444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.166.20:52007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:35827
--ranger_config_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=40015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:21.882822 14444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:21.883127 14444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:21.888196 14450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:21.888188 14449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:21.888362 14444 server_base.cc:1034] running on GCE node
W20260620 22:59:21.888188 14452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:21.888671 14444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:21.888999 14444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:21.890167 14444 hybrid_clock.cc:648] HybridClock initialized: now 1781996361890137 us; error 52 us; skew 500 ppm
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:21.893117 14444 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:59:21.893600 14444 webserver.cc:460] Webserver started at http://127.1.166.62:40015/ using document root <none> and password file <none>
I20260620 22:59:21.893811 14444 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:21.893865 14444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:59:21 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:21.935060 14444 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:59:21.935770 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:21.936070 14444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:59:21.936172 14444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "b5284d3748404d8aa1c41c39e31a9854"
format_stamp: "Formatted at 2026-06-20 22:59:20 on dist-test-slave-rdl9"
server_key: "429b8aac261fe0f8a86b729a178c5ebd"
server_key_iv: "be21de39021fa9015faaaa2260f6f74e"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:21.936614 14444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:21.948082 14444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:21.950314 14444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:21.950526 14444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:21.954533 14444 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:35827
I20260620 22:59:21.954582 14512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:35827 every 8 connection(s)
I20260620 22:59:21.954941 14444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:59:21.956089 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 14444
I20260620 22:59:21.956167 14513 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:59:21.956353 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:52007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:44809
--local_ip_for_outbound_sockets=127.1.166.1
--tserver_master_addrs=127.1.166.62:35827
--webserver_port=35867
--webserver_interface=127.1.166.1
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.166.20:52007/kms with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:59:21.957300 14513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Bootstrap starting.
I20260620 22:59:21.965963 14513 log.cc:824] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:21.967913 14513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:59:21.968192 14513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Bootstrap complete.
I20260620 22:59:21.970960 14513 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } }
I20260620 22:59:21.971218 14513 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5284d3748404d8aa1c41c39e31a9854, State: Initialized, Role: FOLLOWER
I20260620 22:59:21.971374 14513 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } }
I20260620 22:59:21.971444 14513 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:59:21.971493 14513 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:59:21.971539 14513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:59:21.972561 14513 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } }
I20260620 22:59:21.972661 14513 leader_election.cc:302] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5284d3748404d8aa1c41c39e31a9854; no voters:
I20260620 22:59:21.972831 14513 leader_election.cc:288] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:59:21.972879 14517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:59:21.973035 14517 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [term 2 LEADER]: Becoming Leader. State: Replica: b5284d3748404d8aa1c41c39e31a9854, State: Running, Role: LEADER
I20260620 22:59:21.973099 14513 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:21.973219 14517 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } }
I20260620 22:59:21.973574 14519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5284d3748404d8aa1c41c39e31a9854. Latest consensus state: current_term: 2 leader_uuid: "b5284d3748404d8aa1c41c39e31a9854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } } }
I20260620 22:59:21.973665 14519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:21.973843 14524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:21.973583 14518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5284d3748404d8aa1c41c39e31a9854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5284d3748404d8aa1c41c39e31a9854" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 35827 } } }
I20260620 22:59:21.973965 14518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:21.974454 14524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:21.974637 14524 catalog_manager.cc:1259] Loaded cluster ID: 2becfb6924f24763bbefcb55df50b519
I20260620 22:59:21.974680 14524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:59:21.974826 14513 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:59:21.978689 14524 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:59:21.979189 14524 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Loaded TSK: 0
I20260620 22:59:21.979477 14524 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:59:22.075404 14515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:22.075806 14515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:22.075896 14515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:22.077937 14515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:22.078018 14515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:22.078064 14515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:59:22.080571 14515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.166.20:52007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:44809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=35867
--tserver_master_addrs=127.1.166.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:22.080870 14515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:22.081190 14515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:22.090220 14550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:22.090965 14553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:22.091168 14551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:22.091300 14515 server_base.cc:1034] running on GCE node
I20260620 22:59:22.091634 14515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:22.091966 14515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:22.093122 14515 hybrid_clock.cc:648] HybridClock initialized: now 1781996362093099 us; error 38 us; skew 500 ppm
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:22.096501 14515 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:59:22.097086 14515 webserver.cc:460] Webserver started at http://127.1.166.1:35867/ using document root <none> and password file <none>
I20260620 22:59:22.097342 14515 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:22.097427 14515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:22.120543 14515 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260620 22:59:22.122911 14561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.123983 14515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260620 22:59:22.124119 14515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "c3623beec5644a4e8820449d83d8f0cd"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "3002b2b99b4211fe5b0217773e6e50cc"
server_key_iv: "bab2609735a5be0fbbce71cf433bac6c"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:22.124630 14515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:22.145354 14515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:22.147960 14515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:22.148125 14515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:22.148355 14515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:22.148747 14515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:22.148800 14515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.148833 14515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:22.148869 14515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.155817 14515 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:44809
I20260620 22:59:22.155920 14674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:44809 every 8 connection(s)
I20260620 22:59:22.156291 14515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:59:22.165309 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 14515
I20260620 22:59:22.165704 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:52007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:46289
--local_ip_for_outbound_sockets=127.1.166.2
--tserver_master_addrs=127.1.166.62:35827
--webserver_port=39289
--webserver_interface=127.1.166.2
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.166.20:52007/kms with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:22.170816 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:22.157669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35827 (local address 127.1.166.1:39151)
0620 22:59:22.157870 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:22.157932 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:22.158620 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:22.159135 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:22.159140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:22.159405 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:22.159693 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:22.159699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.161238 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.161240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:22.162671 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:22.162675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.163471 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.163473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:22.163632 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:22.164191 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:22.164211 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:22.166288 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:22.168632 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:22.168646 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:22.168649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:22.168943 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:22.169249 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:22.169250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:22.169251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:22.169381 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:22.170001 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:22.170011 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:22.170191 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 22:59:22.170313 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 22:59:22.170558 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260620 22:59:22.170799 14678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:22.157882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:39151 (local address 127.1.166.62:35827)
0620 22:59:22.158073 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:22.158083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:22.158117 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:59:22.158878 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:22.158884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:22.158935 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:22.159058 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:22.159848 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.161082 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.162808 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.163340 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.163437 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:22.166488 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:22.166495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:22.166495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:22.166513 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:22.168503 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:22.169057 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:22.169061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:22.169061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:22.169178 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:22.169474 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:22.169475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:22.169475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:22.169852 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:22.169962 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:22.170450 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 22:59:22.170586 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260620 22:59:22.172282 14675 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35827
I20260620 22:59:22.172405 14675 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.172608 14675 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
W20260620 22:59:22.173616 14675 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:35827 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:59:22.173702 14675 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.173784 14675 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:22.174328 14675 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.174453 14675 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
W20260620 22:59:22.174871 14675 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:59:22.292855 14679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:22.293278 14679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:22.293359 14679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:22.295539 14679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:22.295714 14679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:22.295804 14679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:59:22.298713 14679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.166.20:52007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:46289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=39289
--tserver_master_addrs=127.1.166.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:22.299057 14679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:22.299376 14679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:22.305938 14685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:22.305949 14684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:22.306326 14679 server_base.cc:1034] running on GCE node
W20260620 22:59:22.306164 14687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:22.306612 14679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:22.306967 14679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:22.308159 14679 hybrid_clock.cc:648] HybridClock initialized: now 1781996362308118 us; error 57 us; skew 500 ppm
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:22.312034 14679 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:59:22.312786 14679 webserver.cc:460] Webserver started at http://127.1.166.2:39289/ using document root <none> and password file <none>
I20260620 22:59:22.313120 14679 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:22.313230 14679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:22.340077 14679 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:59:22.341591 14695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.341814 14679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:22.341917 14679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "315832584f5c4e08be508ed19cbf033d"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "a8276e4e830c15a85427faa6d09487d3"
server_key_iv: "e5f4b90dad47e4f219de06a73b4e8e05"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:22.342514 14679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:22.365657 14679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:22.368353 14679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:22.368602 14679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:22.368862 14679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:22.369293 14679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:22.369346 14679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.369380 14679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:22.369436 14679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.377112 14679 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:46289
I20260620 22:59:22.377636 14679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:59:22.378389 14808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:46289 every 8 connection(s)
I20260620 22:59:22.383433 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 14679
I20260620 22:59:22.383904 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.166.20:52007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:37873
--local_ip_for_outbound_sockets=127.1.166.3
--tserver_master_addrs=127.1.166.62:35827
--webserver_port=38601
--webserver_interface=127.1.166.3
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.166.20:52007/kms with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:22.391144 14678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:22.378762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:60061 (local address 127.1.166.62:35827)
0620 22:59:22.378815 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:22.378817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:22.379265 (+ 448us) server_negotiation.cc:407] Connection header received
0620 22:59:22.380005 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:22.380008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:22.380024 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:22.380055 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:22.380959 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.381731 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.383454 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.383783 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.383838 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:22.387430 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:22.387433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:22.387433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:22.387443 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:22.389250 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:22.389977 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:22.389979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:22.389980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:22.390058 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:22.390364 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:22.390365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:22.390366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:22.390509 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:22.390582 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:22.390898 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 22:59:22.391018 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:59:22.391330 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:22.378794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35827 (local address 127.1.166.2:60061)
0620 22:59:22.379055 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:22.379112 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:22.379826 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:22.380253 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:22.380258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:22.380521 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:22.380822 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:22.380828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.381877 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.381878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:22.383296 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:22.383301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.383970 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.383971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:22.384132 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:22.385396 (+ 1264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:22.385408 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:22.387266 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:22.389387 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:22.389402 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:22.389405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:22.389861 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:22.390123 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:22.390124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:22.390125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:22.390264 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:22.390578 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:22.390586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:22.390755 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 22:59:22.390868 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 22:59:22.391045 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":126,"threads_started":1}
I20260620 22:59:22.392280 14809 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35827
I20260620 22:59:22.392468 14809 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.392697 14809 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
W20260620 22:59:22.393528 14809 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:35827 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:59:22.393606 14809 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.393721 14809 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:22.394029 14809 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.394110 14809 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
W20260620 22:59:22.394662 14809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:59:22.517704 14812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:22.518033 14812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:22.518079 14812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:22.520103 14812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:22.520195 14812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:22.520242 14812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:59:22.523766 14812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.166.20:52007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:37873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=38601
--tserver_master_addrs=127.1.166.62:35827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:22.524142 14812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:22.524488 14812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:22.532510 14821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:22.533448 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:22.533969 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:22.534576 14812 server_base.cc:1034] running on GCE node
I20260620 22:59:22.535022 14812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:22.535367 14812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:22.538703 14812 hybrid_clock.cc:648] HybridClock initialized: now 1781996362538564 us; error 156 us; skew 500 ppm
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:22.541661 14812 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:59:22.542207 14812 webserver.cc:460] Webserver started at http://127.1.166.3:38601/ using document root <none> and password file <none>
I20260620 22:59:22.542475 14812 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:22.542547 14812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:22.558423 14812 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:59:22.559963 14829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.560140 14812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:22.560220 14812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "99db9b7da9cf409480a4d801e55d41e5"
format_stamp: "Formatted at 2026-06-20 22:59:21 on dist-test-slave-rdl9"
server_key: "217937721a8554c08558b972121955f6"
server_key_iv: "5fd63d2e05e79656f939b71010385a94"
server_key_version: "kuduclusterkey@0"
I20260620 22:59:22.560753 14812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:22.572593 14812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:22.575265 14812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:22.575435 14812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:22.575656 14812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:22.576026 14812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:22.576064 14812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.576095 14812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:22.576116 14812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:22.583343 14812 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:37873
I20260620 22:59:22.583918 14942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:37873 every 8 connection(s)
I20260620 22:59:22.584327 14812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:59:22.584678 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 14812
Jun 20 22:59:22 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:22.595018 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:22.585892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:59623 (local address 127.1.166.62:35827)
0620 22:59:22.586778 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:22.586780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:22.586785 (+ 5us) server_negotiation.cc:407] Connection header received
0620 22:59:22.586801 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:22.586802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:22.586811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:22.586875 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:22.587385 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.588133 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.589208 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.589403 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.589822 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:22.592028 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:22.592029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:22.592029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:22.592036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:22.593347 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:22.593784 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:22.593793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:22.593793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:22.593854 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:22.594078 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:22.594079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:22.594079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:22.594240 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:22.594298 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:22.594807 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 22:59:22.594906 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":54,"threads_started":1}
I20260620 22:59:22.595237 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:22.585764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:35827 (local address 127.1.166.3:59623)
0620 22:59:22.585949 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:22.585996 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:22.586634 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:22.586958 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:22.586961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:22.587113 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:22.587301 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:22.587305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.588234 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.588235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:22.589130 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:22.589133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.589504 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.589505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:22.589645 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:22.590277 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:22.590287 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:22.591918 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:22.593437 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:22.593448 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:22.593451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:22.593712 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:22.593910 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:22.593911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:22.593911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:22.594023 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:22.594326 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:22.594360 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:22.594641 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 22:59:22.594781 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 22:59:22.594971 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260620 22:59:22.595980 14943 heartbeater.cc:347] Connected to a master server at 127.1.166.62:35827
I20260620 22:59:22.596094 14943 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.596279 14943 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:22.597146 14678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:22.585119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.1.166.62:35827)
0620 22:59:22.585170 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:22.585172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:22.585607 (+ 435us) server_negotiation.cc:407] Connection header received
0620 22:59:22.585730 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:22.585731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:22.585743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:22.585789 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:22.586680 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.587469 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.589037 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:22.589376 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:22.589462 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:22.591704 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:22.591705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:22.591706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:22.591714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:22.594084 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:22.594828 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:22.594830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:22.594830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:22.594899 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:22.595647 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:22.595648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:22.595648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:22.595798 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:22.595974 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:22.596948 (+ 974us) server_negotiation.cc:299] Negotiation successful
0620 22:59:22.597051 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 22:59:22.600011 14943 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:35827 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:59:22.600117 14943 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.600205 14943 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:22.600668 14943 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:22.600780 14943 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
W20260620 22:59:22.601155 14943 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:59:23.175053 14675 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:23.175207 14675 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:23.175787 14477 ts_manager.cc:194] Registered new tserver with Master: c3623beec5644a4e8820449d83d8f0cd (127.1.166.1:44809)
I20260620 22:59:23.177179 14477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:39151
Jun 20 22:59:23 dist-test-slave-rdl9 krb5kdc[13481](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:23.394928 14809 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:23.395136 14809 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:23.395561 14477 ts_manager.cc:194] Registered new tserver with Master: 315832584f5c4e08be508ed19cbf033d (127.1.166.2:46289)
I20260620 22:59:23.396135 14477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:60061
I20260620 22:59:23.601397 14943 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:23.601569 14943 heartbeater.cc:511] Master 127.1.166.62:35827 requested a full tablet report, sending...
I20260620 22:59:23.601871 14477 ts_manager.cc:194] Registered new tserver with Master: 99db9b7da9cf409480a4d801e55d41e5 (127.1.166.3:37873)
I20260620 22:59:23.602451 14477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:59623
I20260620 22:59:23.603138 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:59:23.615219 14965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.604545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.1.166.62:35827)
0620 22:59:23.604700 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:23.604702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:23.604815 (+ 113us) server_negotiation.cc:407] Connection header received
0620 22:59:23.604903 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:23.604904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:23.604921 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:23.605021 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:23.606187 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.607085 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.608449 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.608787 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.608863 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.610518 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:23.610520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:23.610520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:23.610532 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:23.613204 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:23.613746 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:23.613747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:23.613748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:23.613840 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:23.614106 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:23.614107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:23.614108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:23.614295 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:23.614474 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:23.614750 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 22:59:23.614916 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260620 22:59:23.619361 14477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:59:23.622390 14477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:59:23.638976 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.628337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.1.166.1:44809)
0620 22:59:23.632568 (+ 4231us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:23.632572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:23.632586 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:59:23.632639 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:23.632642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:23.632663 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:23.632793 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:23.633216 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.634542 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.636804 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.637728 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.637815 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.637896 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:23.638801 (+ 905us) server_negotiation.cc:299] Negotiation successful
0620 22:59:23.638868 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4195,"thread_start_us":53,"threads_started":1}
I20260620 22:59:23.639798 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.628095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44809 (local address 127.0.0.1:42066)
0620 22:59:23.629297 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:23.629306 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:23.629347 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:23.632978 (+ 3631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:23.632982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:23.632986 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:23.633094 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.633096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.634691 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.634694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.636682 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.636686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.638386 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.638387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.638521 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.638541 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 22:59:23.639579 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0620 22:59:23.639688 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"thread_start_us":95,"threads_started":1}
I20260620 22:59:23.642462 14609 tablet_service.cc:1467] Processing CreateTablet for tablet f75123693a9441f69ad84d3e11c6caac (DEFAULT_TABLE table=test-table [id=7c462cc4563444479b1413bd55eae91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:23.642925 14609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75123693a9441f69ad84d3e11c6caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:23.643576 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.627761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:46289 (local address 127.0.0.1:54224)
0620 22:59:23.629013 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:23.629069 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:23.629168 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:23.638655 (+ 9487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:23.638656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:23.638659 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:23.638706 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.638707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.640055 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.640057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.641293 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.641296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.643291 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.643293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.643394 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.643400 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:59:23.643432 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 22:59:23.643487 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"thread_start_us":165,"threads_started":1}
I20260620 22:59:23.644028 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.628003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.1.166.2:46289)
0620 22:59:23.638476 (+ 10473us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:23.638482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:23.638497 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:59:23.638537 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:23.638539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:23.638559 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:23.638658 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:23.638754 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.639852 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.642244 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.643161 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.643766 (+ 605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.643860 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:23.643893 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 22:59:23.643945 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10415,"thread_start_us":80,"threads_started":1}
I20260620 22:59:23.644335 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.628313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:37873 (local address 127.0.0.1:35212)
0620 22:59:23.635950 (+ 7637us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:23.635963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:23.636026 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:23.636393 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:23.636394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:23.636396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:23.636465 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.636467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.637770 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.637771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.640607 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.640611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.644033 (+ 3422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.644034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.644117 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.644122 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:59:23.644193 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:59:23.644246 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7610,"thread_start_us":71,"threads_started":1}
I20260620 22:59:23.644512 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.628559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.1.166.3:37873)
0620 22:59:23.632471 (+ 3912us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:23.632478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:23.636100 (+ 3622us) server_negotiation.cc:407] Connection header received
0620 22:59:23.636152 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:23.636155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:23.636174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:23.636251 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:23.636542 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.637641 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.640734 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.641626 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.641715 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.641785 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:23.644233 (+ 2448us) server_negotiation.cc:299] Negotiation successful
0620 22:59:23.644318 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3856,"thread_start_us":56,"threads_started":1}
I20260620 22:59:23.645977 14859 tablet_service.cc:1467] Processing CreateTablet for tablet f75123693a9441f69ad84d3e11c6caac (DEFAULT_TABLE table=test-table [id=7c462cc4563444479b1413bd55eae91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:23.646121 14743 tablet_service.cc:1467] Processing CreateTablet for tablet f75123693a9441f69ad84d3e11c6caac (DEFAULT_TABLE table=test-table [id=7c462cc4563444479b1413bd55eae91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:23.646492 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75123693a9441f69ad84d3e11c6caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:23.646492 14859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75123693a9441f69ad84d3e11c6caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:23.651983 14976 tablet_bootstrap.cc:492] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd: Bootstrap starting.
I20260620 22:59:23.653146 14976 tablet_bootstrap.cc:654] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:23.653628 14976 log.cc:824] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:23.654579 14977 tablet_bootstrap.cc:492] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: Bootstrap starting.
I20260620 22:59:23.654873 14978 tablet_bootstrap.cc:492] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: Bootstrap starting.
I20260620 22:59:23.655742 14977 tablet_bootstrap.cc:654] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:23.657188 14978 tablet_bootstrap.cc:654] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:23.657689 14977 log.cc:824] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:23.658139 14978 log.cc:824] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:23.659549 14976 tablet_bootstrap.cc:492] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd: No bootstrap required, opened a new log
I20260620 22:59:23.659644 14976 ts_tablet_manager.cc:1397] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 22:59:23.662506 14976 raft_consensus.cc:348] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.663434 14976 raft_consensus.cc:374] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:23.663555 14976 raft_consensus.cc:729] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3623beec5644a4e8820449d83d8f0cd, State: Initialized, Role: FOLLOWER
I20260620 22:59:23.663815 14976 consensus_queue.cc:260] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.664305 14976 ts_tablet_manager.cc:1428] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260620 22:59:23.664489 14675 heartbeater.cc:503] Master 127.1.166.62:35827 was elected leader, sending a full tablet report...
I20260620 22:59:23.664944 14977 tablet_bootstrap.cc:492] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: No bootstrap required, opened a new log
I20260620 22:59:23.665024 14977 ts_tablet_manager.cc:1397] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
Jun 20 22:59:23 dist-test-slave-rdl9 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for HTTP/127.1.166.20@KRBTEST.COM
I20260620 22:59:23.668095 14977 raft_consensus.cc:348] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.668406 14977 raft_consensus.cc:374] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:23.668496 14977 raft_consensus.cc:729] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315832584f5c4e08be508ed19cbf033d, State: Initialized, Role: FOLLOWER
I20260620 22:59:23.668738 14977 consensus_queue.cc:260] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.669140 14977 ts_tablet_manager.cc:1428] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 22:59:23.669314 14809 heartbeater.cc:503] Master 127.1.166.62:35827 was elected leader, sending a full tablet report...
I20260620 22:59:23.671157 14978 tablet_bootstrap.cc:492] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: No bootstrap required, opened a new log
I20260620 22:59:23.671236 14978 ts_tablet_manager.cc:1397] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260620 22:59:23.673638 14978 raft_consensus.cc:348] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.673807 14978 raft_consensus.cc:374] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:23.673837 14978 raft_consensus.cc:729] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99db9b7da9cf409480a4d801e55d41e5, State: Initialized, Role: FOLLOWER
I20260620 22:59:23.673961 14978 consensus_queue.cc:260] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.674289 14978 ts_tablet_manager.cc:1428] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:59:23.674394 14943 heartbeater.cc:503] Master 127.1.166.62:35827 was elected leader, sending a full tablet report...
I20260620 22:59:23.748399 14982 raft_consensus.cc:482] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:23.748574 14982 raft_consensus.cc:504] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.749019 14982 leader_election.cc:288] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315832584f5c4e08be508ed19cbf033d (127.1.166.2:46289), 99db9b7da9cf409480a4d801e55d41e5 (127.1.166.3:37873)
I20260620 22:59:23.754591 14971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.749961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:39419 (local address 127.1.166.2:46289)
0620 22:59:23.750012 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:23.750014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:23.750646 (+ 632us) server_negotiation.cc:407] Connection header received
0620 22:59:23.750773 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:23.750773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:23.750789 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:23.750821 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:23.751082 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.751798 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.753545 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.754117 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.754169 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.754182 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:23.754459 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 22:59:23.754506 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:59:23.755097 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.749373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:46289 (local address 127.1.166.1:39419)
0620 22:59:23.750561 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:23.750574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:23.750684 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:23.750903 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:23.750904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:23.750906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:23.750992 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.750994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.751924 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.751925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.753425 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.753430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.754258 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.754260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.754365 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.754381 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 22:59:23.754941 (+ 560us) client_negotiation.cc:240] Negotiation successful
0620 22:59:23.754995 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"thread_start_us":121,"threads_started":1}
I20260620 22:59:23.755589 14763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75123693a9441f69ad84d3e11c6caac" candidate_uuid: "c3623beec5644a4e8820449d83d8f0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315832584f5c4e08be508ed19cbf033d" is_pre_election: true
I20260620 22:59:23.755775 14763 raft_consensus.cc:2454] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3623beec5644a4e8820449d83d8f0cd in term 0.
I20260620 22:59:23.756137 14564 leader_election.cc:302] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315832584f5c4e08be508ed19cbf033d, c3623beec5644a4e8820449d83d8f0cd; no voters:
I20260620 22:59:23.756281 14982 raft_consensus.cc:2781] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:59:23.756318 14982 raft_consensus.cc:482] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:59:23.756338 14982 raft_consensus.cc:3037] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:23.757308 14982 raft_consensus.cc:504] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.757458 14982 leader_election.cc:288] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [CANDIDATE]: Term 1 election: Requested vote from peers 315832584f5c4e08be508ed19cbf033d (127.1.166.2:46289), 99db9b7da9cf409480a4d801e55d41e5 (127.1.166.3:37873)
I20260620 22:59:23.757772 14763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75123693a9441f69ad84d3e11c6caac" candidate_uuid: "c3623beec5644a4e8820449d83d8f0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315832584f5c4e08be508ed19cbf033d"
I20260620 22:59:23.757855 14763 raft_consensus.cc:3037] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:23.758649 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.749874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:60043 (local address 127.1.166.3:37873)
0620 22:59:23.749927 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:23.749929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:23.755538 (+ 5609us) server_negotiation.cc:407] Connection header received
0620 22:59:23.755654 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:23.755655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:23.755669 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:23.755697 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:23.755864 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.756519 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.757725 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.758241 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.758290 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.758301 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:23.758523 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 22:59:23.758568 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:59:23.758751 14763 raft_consensus.cc:2454] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3623beec5644a4e8820449d83d8f0cd in term 1.
I20260620 22:59:23.758876 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.749779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:37873 (local address 127.1.166.1:60043)
0620 22:59:23.755483 (+ 5704us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:23.755494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:23.755568 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:23.755759 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:23.755760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:23.755761 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:23.755796 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.755797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.756625 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.756625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.757625 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:23.757628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.758380 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.758380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:23.758442 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.758445 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:59:23.758761 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 22:59:23.758807 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5689,"thread_start_us":93,"threads_started":1}
I20260620 22:59:23.759298 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75123693a9441f69ad84d3e11c6caac" candidate_uuid: "c3623beec5644a4e8820449d83d8f0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99db9b7da9cf409480a4d801e55d41e5"
I20260620 22:59:23.759408 14896 raft_consensus.cc:3037] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:23.760200 14896 raft_consensus.cc:2454] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3623beec5644a4e8820449d83d8f0cd in term 1.
I20260620 22:59:23.760376 14897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75123693a9441f69ad84d3e11c6caac" candidate_uuid: "c3623beec5644a4e8820449d83d8f0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99db9b7da9cf409480a4d801e55d41e5" is_pre_election: true
I20260620 22:59:23.760449 14897 raft_consensus.cc:2362] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c3623beec5644a4e8820449d83d8f0cd in term 1. Re-sending same reply.
I20260620 22:59:23.760607 14562 leader_election.cc:302] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99db9b7da9cf409480a4d801e55d41e5, c3623beec5644a4e8820449d83d8f0cd; no voters:
I20260620 22:59:23.760865 14982 raft_consensus.cc:2781] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:23.761044 14982 raft_consensus.cc:686] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [term 1 LEADER]: Becoming Leader. State: Replica: c3623beec5644a4e8820449d83d8f0cd, State: Running, Role: LEADER
I20260620 22:59:23.761137 14982 consensus_queue.cc:237] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } }
I20260620 22:59:23.762302 14477 catalog_manager.cc:5495] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd reported cstate change: term changed from 0 to 1, leader changed from <none> to c3623beec5644a4e8820449d83d8f0cd (127.1.166.1). New cstate: current_term: 1 leader_uuid: "c3623beec5644a4e8820449d83d8f0cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3623beec5644a4e8820449d83d8f0cd" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 44809 } health_report { overall_health: HEALTHY } } }
I20260620 22:59:23.784854 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.781728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.1.166.1:44809)
0620 22:59:23.781776 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:23.781778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:23.781788 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:59:23.781838 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:23.781839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:23.781869 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:23.781962 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:59:23.782103 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.782780 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.783939 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:23.784238 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:23.784320 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:23.784482 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:59:23.784633 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:59:23.784718 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:23.784735 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 22:59:23.784760 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:59:23.787001 14897 raft_consensus.cc:1261] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 1 FOLLOWER]: Refusing update from remote peer c3623beec5644a4e8820449d83d8f0cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:23.787011 14763 raft_consensus.cc:1261] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 1 FOLLOWER]: Refusing update from remote peer c3623beec5644a4e8820449d83d8f0cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:23.787369 14987 consensus_queue.cc:1035] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "99db9b7da9cf409480a4d801e55d41e5" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:23.787662 14982 consensus_queue.cc:1035] T f75123693a9441f69ad84d3e11c6caac P c3623beec5644a4e8820449d83d8f0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "315832584f5c4e08be508ed19cbf033d" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 46289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:23.795922 14990 mvcc.cc:205] Tried to move back new op lower bound from 7299057106068647936 to 7299057105966702592. Current Snapshot: MvccSnapshot[applied={T|T < 7299057106068647936}]
I20260620 22:59:23.798714 14992 mvcc.cc:205] Tried to move back new op lower bound from 7299057106068647936 to 7299057105966702592. Current Snapshot: MvccSnapshot[applied={T|T < 7299057106068647936}]
I20260620 22:59:23.799058 14991 mvcc.cc:205] Tried to move back new op lower bound from 7299057106068647936 to 7299057105966702592. Current Snapshot: MvccSnapshot[applied={T|T < 7299057106068647936}]
I20260620 22:59:23.799053 14476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:59:23.799170 14476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:59:23.800657 14476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5284d3748404d8aa1c41c39e31a9854: Sending DeleteTablet for 3 replicas of tablet f75123693a9441f69ad84d3e11c6caac
I20260620 22:59:23.801108 14859 tablet_service.cc:1514] Processing DeleteTablet for tablet f75123693a9441f69ad84d3e11c6caac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:23 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:35212
I20260620 22:59:23.801690 14743 tablet_service.cc:1514] Processing DeleteTablet for tablet f75123693a9441f69ad84d3e11c6caac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:23 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:54224
I20260620 22:59:23.801946 14609 tablet_service.cc:1514] Processing DeleteTablet for tablet f75123693a9441f69ad84d3e11c6caac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:23 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:42066
I20260620 22:59:23.803144 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 14515
I20260620 22:59:23.805564 15000 tablet_replica.cc:330] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: stopping tablet replica
I20260620 22:59:23.805759 15000 raft_consensus.cc:2229] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:59:23.805907 15000 raft_consensus.cc:2258] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:23.806123 14999 tablet_replica.cc:330] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: stopping tablet replica
I20260620 22:59:23.806268 14999 raft_consensus.cc:2229] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:59:23.806411 14999 raft_consensus.cc:2258] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:23.806921 14999 ts_tablet_manager.cc:1905] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:59:23.809124 14999 ts_tablet_manager.cc:1918] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:59:23.809255 14999 log.cc:1196] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/f75123693a9441f69ad84d3e11c6caac
I20260620 22:59:23.809520 14999 ts_tablet_manager.cc:1939] T f75123693a9441f69ad84d3e11c6caac P 99db9b7da9cf409480a4d801e55d41e5: Deleting consensus metadata
I20260620 22:59:23.809690 15000 ts_tablet_manager.cc:1905] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:59:23.810053 14461 catalog_manager.cc:4841] TS 99db9b7da9cf409480a4d801e55d41e5 (127.1.166.3:37873): tablet f75123693a9441f69ad84d3e11c6caac (table test-table [id=7c462cc4563444479b1413bd55eae91c]) successfully deleted
I20260620 22:59:23.811918 15000 ts_tablet_manager.cc:1918] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:59:23.812080 15000 log.cc:1196] T f75123693a9441f69ad84d3e11c6caac P 315832584f5c4e08be508ed19cbf033d: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/f75123693a9441f69ad84d3e11c6caac
W20260620 22:59:23.812641 14463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:44809 (error 108)
I20260620 22:59:23.812819 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 14679
I20260620 22:59:23.815800 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.815628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:44809 (local address 127.0.0.1:42084)
0620 22:59:23.815697 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:23.815740 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 22:59:23.815945 14463 catalog_manager.cc:4584] TS c3623beec5644a4e8820449d83d8f0cd (127.1.166.1:44809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f75123693a9441f69ad84d3e11c6caac: Network error: Client connection negotiation failed: client connection to 127.1.166.1:44809: connect: Connection refused (error 111)
I20260620 22:59:23.821033 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:23.820905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:46289 (local address 127.0.0.1:54234)
0620 22:59:23.820954 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:23.820985 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.2:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 22:59:23.821074 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 14812
I20260620 22:59:23.828897 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 14444
I20260620 22:59:23.835127 1688 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 22:59:24.171100 1688 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 22:59:24.171203 1688 mini_ranger.cc:66] Stopping Ranger...
I20260620 22:59:24.517323 1688 mini_ranger.cc:68] Stopped Ranger
2026-06-20 22:59:24.517 UTC [13511] LOG: received smart shutdown request
2026-06-20 22:59:24.521 UTC [13511] LOG: background worker "logical replication launcher" (PID 13520) exited with exit code 1
2026-06-20 22:59:24.521 UTC [13515] LOG: shutting down
2026-06-20 22:59:24.654 UTC [13511] LOG: database system is shut down
2026-06-20T22:59:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53507 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:59:24 dist-test-slave-rdl9 krb5kdc[15011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:59:24 dist-test-slave-rdl9 krb5kdc[15011](info): set up 2 sockets
Jun 20 22:59:24 dist-test-slave-rdl9 krb5kdc[15011](info): commencing operation
krb5kdc: starting...
W20260620 22:59:25.726317 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:59:25 dist-test-slave-rdl9 krb5kdc[15011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:59:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:59:25.849313 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:33775
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:33775,127.1.166.61:44131,127.1.166.60:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:25.939790 15024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:25.940055 15024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:25.940095 15024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:25.941452 15024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:25.941540 15024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:25.941567 15024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:25.941587 15024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:25.941607 15024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:25.943476 15024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:33775,127.1.166.61:44131,127.1.166.60:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:25.943804 15024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:25.944046 15024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:25.949338 15030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:25.949494 15024 server_base.cc:1034] running on GCE node
W20260620 22:59:25.949359 15032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:25.949343 15029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:25.949911 15024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:25.950246 15024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:25.951424 15024 hybrid_clock.cc:648] HybridClock initialized: now 1781996365951379 us; error 64 us; skew 500 ppm
Jun 20 22:59:25 dist-test-slave-rdl9 krb5kdc[15011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:25.957808 15024 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:59:25.958429 15024 webserver.cc:460] Webserver started at http://127.1.166.62:36921/ using document root <none> and password file <none>
I20260620 22:59:25.958700 15024 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:25.958752 15024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:25.958871 15024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:25.960386 15024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "67b62a4f5c4c4fe88ced648f6ba87f63"
format_stamp: "Formatted at 2026-06-20 22:59:25 on dist-test-slave-rdl9"
server_key: "920fcc6be8c6bb542c7ac68767629891"
server_key_iv: "b3190e43cb9da9ff3fad25e87cafcae7"
server_key_version: "encryptionkey@0"
I20260620 22:59:25.960836 15024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "67b62a4f5c4c4fe88ced648f6ba87f63"
format_stamp: "Formatted at 2026-06-20 22:59:25 on dist-test-slave-rdl9"
server_key: "920fcc6be8c6bb542c7ac68767629891"
server_key_iv: "b3190e43cb9da9ff3fad25e87cafcae7"
server_key_version: "encryptionkey@0"
I20260620 22:59:25.962278 15024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 22:59:25.963253 15039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:25.963577 15024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:59:25.963675 15024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "67b62a4f5c4c4fe88ced648f6ba87f63"
format_stamp: "Formatted at 2026-06-20 22:59:25 on dist-test-slave-rdl9"
server_key: "920fcc6be8c6bb542c7ac68767629891"
server_key_iv: "b3190e43cb9da9ff3fad25e87cafcae7"
server_key_version: "encryptionkey@0"
I20260620 22:59:25.963762 15024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:25.973223 15024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:25.976229 15024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:25.976413 15024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:25.980587 15024 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:33775
I20260620 22:59:25.980654 15091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:33775 every 8 connection(s)
I20260620 22:59:25.981002 15024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:59:25.981931 15092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:25.983644 15092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:59:25.985167 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15024
I20260620 22:59:25.985307 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:59:25.985529 1688 external_mini_cluster.cc:1428] Setting key b825e641c2ec917e0650ecad4d48b2bb
Jun 20 22:59:25 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.61@KRBTEST.COM; defaulting to no policy
I20260620 22:59:25.998034 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:25.984534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33775 (local address 127.0.0.1:43170)
0620 22:59:25.984774 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:25.984822 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:25.985687 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:25.986214 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:25.986218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:25.986715 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:25.987013 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:25.987021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:25.988249 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:25.988251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:25.989058 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:25.989061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:25.989796 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:25.989797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:25.989889 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:25.990579 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:25.990593 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:25.992752 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:25.996303 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:25.996312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:25.996315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:25.996546 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:25.996850 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:25.996851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:25.996851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:25.996953 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:25.997403 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:25.997410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:25.997479 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 22:59:25.997565 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 22:59:25.997784 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":120,"threads_started":1}
I20260620 22:59:25.998045 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:25.984577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.1.166.62:33775)
0620 22:59:25.984818 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:25.984843 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:25.984916 (+ 73us) server_negotiation.cc:407] Connection header received
0620 22:59:25.985998 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:25.986002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:25.986033 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:25.986175 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:25.987179 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:25.988094 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:25.989168 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:25.989637 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:25.989775 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:25.992883 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:25.992896 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:25.992896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:25.992907 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:25.996144 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:25.996646 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:25.996650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:25.996650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:25.996773 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:25.997008 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:25.997009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:25.997010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:25.997304 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:25.997375 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:25.997646 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 22:59:25.997877 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":202,"thread_start_us":136,"threads_started":1}
I20260620 22:59:25.999142 15092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.61@KRBTEST.COM" created.
I20260620 22:59:25.999553 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:25.999423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50408)
0620 22:59:25.999477 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:25.999528 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:59:25.999681 15042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
I20260620 22:59:26.000391 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.000228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50422)
0620 22:59:26.000320 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.000365 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 22:59:26.000550 15092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:44131: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab.
Entry for principal kudu/127.1.166.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab.
I20260620 22:59:26.017028 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.61:44131
--webserver_interface=127.1.166.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab
--principal=kudu/127.1.166.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:33775,127.1.166.61:44131,127.1.166.60:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:59:26.049945 15092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } attempt: 1
I20260620 22:59:26.050609 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.050455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50438)
0620 22:59:26.050523 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.050573 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 22:59:26.051120 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.050973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50450)
0620 22:59:26.051051 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.051086 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 22:59:26.051254 15092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:44131: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
W20260620 22:59:26.109968 15099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:26.110239 15099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:26.110281 15099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:26.112008 15099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:26.112128 15099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:26.112156 15099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:26.112185 15099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:26.112201 15099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:26.114725 15099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:33775,127.1.166.61:44131,127.1.166.60:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab
--principal=kudu/127.1.166.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.61:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.166.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:26.115044 15099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:26.115294 15099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:59:26.119405 15092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } attempt: 2
I20260620 22:59:26.119876 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.119746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50458)
0620 22:59:26.119806 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.119839 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 22:59:26.120240 15107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:26.120317 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.120170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50474)
0620 22:59:26.120216 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.120237 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 22:59:26.120250 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:26.120452 15092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:44131: Network error: Client connection negotiation failed: client connection to 127.1.166.61:44131: connect: Connection refused (error 111)
W20260620 22:59:26.120333 15104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:26.120623 15099 server_base.cc:1034] running on GCE node
I20260620 22:59:26.120862 15099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:26.121182 15099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:26.122326 15099 hybrid_clock.cc:648] HybridClock initialized: now 1781996366122308 us; error 29 us; skew 500 ppm
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:26.126076 15099 init.cc:377] Logged in from keytab as kudu/127.1.166.61@KRBTEST.COM (short username kudu)
I20260620 22:59:26.126757 15099 webserver.cc:460] Webserver started at http://127.1.166.61:37523/ using document root <none> and password file <none>
I20260620 22:59:26.126999 15099 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:26.127063 15099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:26.127218 15099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:26.128476 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/data/instance:
uuid: "80cd152c055e47a38eb40eaf678df6c6"
format_stamp: "Formatted at 2026-06-20 22:59:26 on dist-test-slave-rdl9"
server_key: "1b4a3d9d7959d48992106d7d9b47eb17"
server_key_iv: "afc97135759c5d564544f232263b49a1"
server_key_version: "encryptionkey@0"
I20260620 22:59:26.128894 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/wal/instance:
uuid: "80cd152c055e47a38eb40eaf678df6c6"
format_stamp: "Formatted at 2026-06-20 22:59:26 on dist-test-slave-rdl9"
server_key: "1b4a3d9d7959d48992106d7d9b47eb17"
server_key_iv: "afc97135759c5d564544f232263b49a1"
server_key_version: "encryptionkey@0"
I20260620 22:59:26.130749 15099 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 22:59:26.131817 15114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:26.132071 15099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:26.132186 15099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/wal
uuid: "80cd152c055e47a38eb40eaf678df6c6"
format_stamp: "Formatted at 2026-06-20 22:59:26 on dist-test-slave-rdl9"
server_key: "1b4a3d9d7959d48992106d7d9b47eb17"
server_key_iv: "afc97135759c5d564544f232263b49a1"
server_key_version: "encryptionkey@0"
I20260620 22:59:26.132284 15099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:26.149192 15099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:26.151871 15099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:26.152037 15099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:26.156225 15099 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.61:44131
I20260620 22:59:26.156260 15166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.61:44131 every 8 connection(s)
I20260620 22:59:26.156665 15099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
I20260620 22:59:26.157626 15167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:26.159512 15167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:59:26.163089 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15099
I20260620 22:59:26.163229 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-1/wal/instance
I20260620 22:59:26.163604 1688 external_mini_cluster.cc:1428] Setting key 316017b75373fea3b83a4757b16dc13d
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.60@KRBTEST.COM; defaulting to no policy
I20260620 22:59:26.173219 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.160753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.1.166.62:33775)
0620 22:59:26.160938 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.160941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.161206 (+ 265us) server_negotiation.cc:407] Connection header received
0620 22:59:26.162071 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.162073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.162119 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.162174 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.163582 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.164138 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.166059 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.166297 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.166415 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.169229 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.169232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.169232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.169241 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.171079 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.171757 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.171758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.171759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.171842 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.172216 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.172217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.172217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.172370 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.172467 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.172968 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.173097 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 22:59:26.173408 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.160670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33775 (local address 127.0.0.1:43182)
0620 22:59:26.161039 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.161094 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:26.161897 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:26.162278 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:26.162283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:26.162728 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:26.163264 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.163279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.164291 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.164293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.165907 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.165916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.166437 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.166438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.166605 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.167194 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:26.167205 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:26.169066 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:26.171197 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.171206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.171209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.171644 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.171934 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.171936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.171936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.172087 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.172474 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:26.172480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:26.172703 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 22:59:26.172848 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 22:59:26.173084 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":179,"threads_started":1}
I20260620 22:59:26.174258 15167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.60@KRBTEST.COM" created.
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
I20260620 22:59:26.184645 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.174817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50490)
0620 22:59:26.174887 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.174902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:26.174964 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:26.175288 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:26.175289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:26.175502 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:26.175586 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.175589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.176674 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.176675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.177539 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.177544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.178329 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.178331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.178496 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.179008 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:26.179013 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:26.180804 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:26.183196 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.183198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.183208 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.183438 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.183783 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.183784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.183785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.183856 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.184288 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:26.184289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:26.184352 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 22:59:26.184433 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 22:59:26.184549 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:59:26.184823 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.174904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.1.166.61:44131)
0620 22:59:26.175126 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.175131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.175151 (+ 20us) server_negotiation.cc:407] Connection header received
0620 22:59:26.175183 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.175185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.175200 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.175274 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.175711 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.176566 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.177649 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.178190 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.178280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.180968 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.180983 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.180983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.180992 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.183080 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.183574 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.183578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.183579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.183675 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.183982 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.183983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.183983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.184205 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.184287 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.184529 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.184640 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260620 22:59:26.185729 15167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:59:26.186142 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.186022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54904)
0620 22:59:26.186069 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.186117 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 22:59:26.186303 15115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
I20260620 22:59:26.186758 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.186671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54912)
0620 22:59:26.186696 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.186722 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 22:59:26.186872 15167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:40069: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab.
Entry for principal kudu/127.1.166.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab.
I20260620 22:59:26.192802 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.60:40069
--webserver_interface=127.1.166.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab
--principal=kudu/127.1.166.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:33775,127.1.166.61:44131,127.1.166.60:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:59:26.211666 15092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } attempt: 3
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
I20260620 22:59:26.220945 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.212218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50496)
0620 22:59:26.212342 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.212360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:26.212467 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:26.212668 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:26.212669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:26.212873 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:26.212954 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.212955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.214209 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.214211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.214845 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.214848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.215409 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.215411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.215554 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.216109 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:26.216115 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:26.217492 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:26.219481 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.219482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.219483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.219872 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.220215 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.220216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.220217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.220316 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.220636 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:26.220637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:26.220695 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 22:59:26.220747 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:59:26.220866 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 22:59:26.220990 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.212344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.1.166.61:44131)
0620 22:59:26.212447 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.212450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.212469 (+ 19us) server_negotiation.cc:407] Connection header received
0620 22:59:26.212554 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.212556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.212575 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.212653 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.213038 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.214025 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.214943 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.215294 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.215377 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.217602 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.217603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.217604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.217614 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.219353 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.219995 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.219997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.219997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.220096 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.220395 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.220395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.220396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.220559 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.220650 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.220785 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.220879 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 22:59:26.221463 15092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:59:26.221853 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.221748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54926)
0620 22:59:26.221795 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.221828 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:59:26.222218 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.222092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54930)
0620 22:59:26.222160 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.222194 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 22:59:26.222316 15092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:40069: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
I20260620 22:59:26.236148 15167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } attempt: 1
I20260620 22:59:26.236765 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.236570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54932)
0620 22:59:26.236672 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.236731 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 22:59:26.237149 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.237049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54948)
0620 22:59:26.237095 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.237129 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:59:26.237236 15167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:40069: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
I20260620 22:59:26.253535 15092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } attempt: 1
I20260620 22:59:26.254254 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.254057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54960)
0620 22:59:26.254160 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.254202 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 22:59:26.254863 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.254606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54968)
0620 22:59:26.254741 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.254811 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 22:59:26.255045 15092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:40069: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
W20260620 22:59:26.287948 15174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:26.288214 15174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:26.288252 15174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:26.289580 15174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:26.289650 15174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:26.289673 15174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:26.289692 15174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:26.289710 15174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:26.291477 15174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:33775,127.1.166.61:44131,127.1.166.60:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab
--principal=kudu/127.1.166.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.60:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.166.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:26.291811 15174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:26.292058 15174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:59:26.297477 15174 server_base.cc:1034] running on GCE node
W20260620 22:59:26.297430 15179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:26.297442 15182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:26.297417 15180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:26.298122 15174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:26.298517 15174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:26.299685 15174 hybrid_clock.cc:648] HybridClock initialized: now 1781996366299666 us; error 32 us; skew 500 ppm
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:26.302770 15174 init.cc:377] Logged in from keytab as kudu/127.1.166.60@KRBTEST.COM (short username kudu)
I20260620 22:59:26.303448 15174 webserver.cc:460] Webserver started at http://127.1.166.60:39829/ using document root <none> and password file <none>
I20260620 22:59:26.303699 15174 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:26.303753 15174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:26.303905 15174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:26.305408 15167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } attempt: 2
I20260620 22:59:26.305560 15174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/data/instance:
uuid: "ee5b5c4fcc10411c8be95d2f12fefac5"
format_stamp: "Formatted at 2026-06-20 22:59:26 on dist-test-slave-rdl9"
server_key: "f82a88a452549837d7916c3707dcbf9c"
server_key_iv: "8ee55b980c66658d3d26aedd2ac923a8"
server_key_version: "encryptionkey@0"
I20260620 22:59:26.306074 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.305789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54976)
0620 22:59:26.305917 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.306038 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 22:59:26.306401 15174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/wal/instance:
uuid: "ee5b5c4fcc10411c8be95d2f12fefac5"
format_stamp: "Formatted at 2026-06-20 22:59:26 on dist-test-slave-rdl9"
server_key: "f82a88a452549837d7916c3707dcbf9c"
server_key_iv: "8ee55b980c66658d3d26aedd2ac923a8"
server_key_version: "encryptionkey@0"
I20260620 22:59:26.306594 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.306493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54978)
0620 22:59:26.306540 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.306568 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 22:59:26.306703 15167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:40069: Network error: Client connection negotiation failed: client connection to 127.1.166.60:40069: connect: Connection refused (error 111)
I20260620 22:59:26.308434 15174 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 22:59:26.309517 15189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:26.309772 15174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:26.309880 15174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/wal
uuid: "ee5b5c4fcc10411c8be95d2f12fefac5"
format_stamp: "Formatted at 2026-06-20 22:59:26 on dist-test-slave-rdl9"
server_key: "f82a88a452549837d7916c3707dcbf9c"
server_key_iv: "8ee55b980c66658d3d26aedd2ac923a8"
server_key_version: "encryptionkey@0"
I20260620 22:59:26.309978 15174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:26.320829 15174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:26.323503 15174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:26.323660 15174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:26.328078 15174 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.60:40069
I20260620 22:59:26.328168 15241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.60:40069 every 8 connection(s)
I20260620 22:59:26.328490 15174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
I20260620 22:59:26.329113 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15174
I20260620 22:59:26.329200 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996175763767-1688-0/minicluster-data/master-2/wal/instance
I20260620 22:59:26.329384 1688 external_mini_cluster.cc:1428] Setting key d200a28e787eb21dfdbb461d2df695b6
I20260620 22:59:26.329375 15242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:26.330191 15092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } attempt: 2
I20260620 22:59:26.331996 15242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:26.341845 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.330785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54992)
0620 22:59:26.330844 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.330854 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:26.330913 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:26.331369 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:26.331370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:26.331566 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:26.331616 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.331618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.333231 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.333232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.334080 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.334083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.335008 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.335008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.335073 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.335623 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:26.335628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:26.337477 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:26.339657 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.339659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.339659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.339843 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.340441 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.340442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.340443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.340509 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.341408 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:26.341409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:26.341466 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 22:59:26.341616 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 22:59:26.341729 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:59:26.342247 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.330565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.1.166.60:40069)
0620 22:59:26.330686 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.330687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.330910 (+ 223us) server_negotiation.cc:407] Connection header received
0620 22:59:26.331124 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.331131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.331202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.331681 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.331727 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.333111 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.334190 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.334924 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.335033 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.337581 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.337590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.337590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.337598 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.339583 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.339954 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.339959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.339959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.340387 (+ 428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.341175 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.341176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.341176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.341325 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.341390 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.341834 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.341997 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260620 22:59:26.342267 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.332716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.1.166.62:33775)
0620 22:59:26.332778 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.332780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.332943 (+ 163us) server_negotiation.cc:407] Connection header received
0620 22:59:26.333088 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.333089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.333101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.333152 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.333617 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.334091 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.334959 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.335218 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.335281 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.339533 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.339535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.339535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.339543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.341094 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.341501 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.341502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.341502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.341556 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.341750 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.341750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.341751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.341876 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.341932 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.342097 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.342178 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:59:26.342537 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.332631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:33775 (local address 127.0.0.1:43194)
0620 22:59:26.332842 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.332880 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:26.332947 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:26.333176 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:26.333178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:26.333473 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:26.333540 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.333542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.334161 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.334162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.334874 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.334877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.335305 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.335306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.335386 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.336119 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:26.336130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:26.339423 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:26.341176 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.341183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.341185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.341422 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.341617 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.341618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.341618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.341687 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.341929 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:26.341936 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:26.341989 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 22:59:26.342063 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:59:26.342181 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
I20260620 22:59:26.342701 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.330028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.1.166.60:40069)
0620 22:59:26.330240 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.330250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.330285 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:59:26.331121 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.331131 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.331202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.331356 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.332069 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.333111 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.334191 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.334923 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.335033 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.338202 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.338203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.338204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.338212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.340280 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.340728 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.340729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.340729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.340783 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.340956 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.340957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.340958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.341325 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.341443 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.342436 (+ 993us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.342536 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":253,"server-negotiator.queue_time_us":155,"thread_start_us":70,"threads_started":1}
I20260620 22:59:26.343083 15242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:59:26.344244 15092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63: Bootstrap starting.
I20260620 22:59:26.345088 15092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:26.345387 15092 log.cc:824] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:26.346690 15092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63: No bootstrap required, opened a new log
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:26.348938 15092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.349145 15092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:26.349184 15092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b62a4f5c4c4fe88ced648f6ba87f63, State: Initialized, Role: FOLLOWER
I20260620 22:59:26.349344 15092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.349743 15092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:26.349686 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.349784 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:59:26.351646 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.343357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:44131 (local address 127.0.0.1:50500)
0620 22:59:26.343456 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.343466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:26.343503 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:26.343695 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:26.343697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:26.343863 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:26.343930 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.343933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.344732 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.344733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.345314 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.345316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.345827 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.345828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.345899 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.346390 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:26.346396 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:26.348041 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:26.350126 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.350129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.350129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.350382 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.350613 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.350614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.350614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.350679 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.351240 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:26.351242 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:26.351316 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 22:59:26.351400 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:59:26.351531 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 22:59:26.351696 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.343461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.1.166.61:44131)
0620 22:59:26.343523 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.343525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.343535 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:59:26.343562 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.343562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.343575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.343637 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.344023 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.344654 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.345402 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.345685 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.345761 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.348214 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.348217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.348218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.348230 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.350017 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.350478 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.350479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.350480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.350545 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.350736 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.350737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.350738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.351050 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.351177 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.351483 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.351596 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:59:26.352221 15242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
W20260620 22:59:26.352277 15260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:59:26.352322 15260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:59:26.354806 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.343657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.1.166.62:33775)
0620 22:59:26.343689 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.343691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.343731 (+ 40us) server_negotiation.cc:407] Connection header received
0620 22:59:26.343786 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.343787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.343795 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.343838 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.344412 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.345071 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.346039 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.346247 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.346299 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.349506 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.349508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.349508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.349519 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.353076 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.353572 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.353573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.353573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.353645 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.353850 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.353851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.353851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.353985 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.354148 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.354611 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.354706 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
I20260620 22:59:26.352698 15024 master_runner.cc:186] Error getting master registration for 127.1.166.61:44131: OK, instance_id { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" instance_seqno: 1781996366154919 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:59:26.357040 15024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:59:26.361541 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.352498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:54994)
0620 22:59:26.352558 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.352570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:26.352628 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:26.352756 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:26.352757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:26.352911 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:26.352950 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.352953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.353790 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.353791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.354522 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.354526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.355053 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.355054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.355097 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.355528 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:26.355534 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:26.357019 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:26.359420 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.359421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.359422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.359673 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.360576 (+ 903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.360577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.360577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.360653 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.361183 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:26.361184 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:26.361229 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:59:26.361294 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 22:59:26.361418 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:59:26.361596 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.352537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.1.166.60:40069)
0620 22:59:26.352574 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.352576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.352611 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:59:26.352684 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.352685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.352696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.352753 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.353032 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.353706 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.354607 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.354911 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.354987 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.357148 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.357149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.357150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.357158 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.359186 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.360418 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.360419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.360419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.360500 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.360949 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.360949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.360950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.361128 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.361195 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.361326 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.361488 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":20}
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
I20260620 22:59:26.363305 15242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5: Bootstrap starting.
I20260620 22:59:26.364091 15242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:26.364473 15242 log.cc:824] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:26.366107 15242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5: No bootstrap required, opened a new log
I20260620 22:59:26.367905 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.357734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.1.166.61:44131)
0620 22:59:26.357810 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.357813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.357821 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:59:26.357844 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.357846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.357857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.357910 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.358797 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.359674 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.360397 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.360663 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.360724 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.363862 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.363864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.363864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.363874 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.366060 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.366607 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.366608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.366609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.366679 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.366884 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.366884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.366885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.367029 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.367224 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.367706 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.367797 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 22:59:26.368472 15242 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.368656 15242 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:26.368721 15242 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5b5c4fcc10411c8be95d2f12fefac5, State: Initialized, Role: FOLLOWER
I20260620 22:59:26.368866 15242 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.369289 15242 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:26.369220 15263 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.369798 15263 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:59:26.369817 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 22:59:26.371191 15274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:59:26.371276 15274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:59:26.372887 15174 master_runner.cc:186] Error getting master registration for 127.1.166.61:44131: OK, instance_id { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" instance_seqno: 1781996366154919 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:59:26.375192 15174 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:59:26.376425 15263 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:26.376511 15263 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.377120 15066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" is_pre_election: true
I20260620 22:59:26.377265 15066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5b5c4fcc10411c8be95d2f12fefac5 in term 0.
I20260620 22:59:26.377449 15190 leader_election.cc:302] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b62a4f5c4c4fe88ced648f6ba87f63, ee5b5c4fcc10411c8be95d2f12fefac5; no voters:
I20260620 22:59:26.377974 15263 leader_election.cc:288] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b62a4f5c4c4fe88ced648f6ba87f63 (127.1.166.62:33775), 80cd152c055e47a38eb40eaf678df6c6 (127.1.166.61:44131)
I20260620 22:59:26.378041 15263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:59:26.378094 15263 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:59:26.378118 15263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:26.379702 15263 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.379925 15263 leader_election.cc:288] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [CANDIDATE]: Term 1 election: Requested vote from peers 67b62a4f5c4c4fe88ced648f6ba87f63 (127.1.166.62:33775), 80cd152c055e47a38eb40eaf678df6c6 (127.1.166.61:44131)
I20260620 22:59:26.380822 15066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63"
I20260620 22:59:26.381091 15066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:26.382262 15066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5b5c4fcc10411c8be95d2f12fefac5 in term 1.
I20260620 22:59:26.382997 15190 leader_election.cc:302] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b62a4f5c4c4fe88ced648f6ba87f63, ee5b5c4fcc10411c8be95d2f12fefac5; no voters:
I20260620 22:59:26.383229 15263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:26.383308 15263 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [term 1 LEADER]: Becoming Leader. State: Replica: ee5b5c4fcc10411c8be95d2f12fefac5, State: Running, Role: LEADER
I20260620 22:59:26.383327 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.371954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.1.166.61:44131)
0620 22:59:26.371991 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.371992 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.374712 (+ 2720us) server_negotiation.cc:407] Connection header received
0620 22:59:26.374798 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.374799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.374809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.374836 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.375600 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.376379 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.377417 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.377797 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.377851 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.379545 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.379546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.379547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.379555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.381259 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.382123 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.382125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.382125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.382206 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.382469 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.382470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.382470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.382744 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.382845 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.383124 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.383219 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:59:26.383419 15263 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.383950 15263 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee5b5c4fcc10411c8be95d2f12fefac5. Latest consensus state: current_term: 1 leader_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.384061 15263 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:26.387079 15285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:26.387343 15285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:26.390026 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.371683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.1.166.62:33775)
0620 22:59:26.371732 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.371734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.374558 (+ 2824us) server_negotiation.cc:407] Connection header received
0620 22:59:26.374627 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.374628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.374640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.374691 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.383176 (+ 8485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.383836 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.384984 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.385298 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.385346 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.387040 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.387042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.387042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.387050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.388576 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.389027 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.389028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.389028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.389102 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.389330 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.389331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.389331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.389480 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.389584 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.389815 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.389912 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:59:26.390400 15066 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [term 1 FOLLOWER]: Refusing update from remote peer ee5b5c4fcc10411c8be95d2f12fefac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:26.391193 15283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:26.393007 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.372175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.1.166.60:40069)
0620 22:59:26.372224 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.372226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.383665 (+ 11439us) server_negotiation.cc:407] Connection header received
0620 22:59:26.383772 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.383773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.383786 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.383815 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.384490 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.385223 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.386212 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.386575 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.386663 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.389195 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.389197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.389197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.389205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.390832 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.391272 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.391273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.391274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.391365 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.391993 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.391995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.391995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.392149 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.392235 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.392792 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.392912 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:59:26.392975 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee5b5c4fcc10411c8be95d2f12fefac5. Latest consensus state: current_term: 1 leader_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.393092 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:59:26.397851 15167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } attempt: 3
I20260620 22:59:26.399833 15263 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.399960 15263 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:26.400063 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.400146 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:59:26.400339 15286 mvcc.cc:205] Tried to move back new op lower bound from 7299057116728623104 to 7299057116707430400. Current Snapshot: MvccSnapshot[applied={T|T < 7299057116728623104}]
I20260620 22:59:26.400635 15287 mvcc.cc:205] Tried to move back new op lower bound from 7299057116728623104 to 7299057116707430400. Current Snapshot: MvccSnapshot[applied={T|T < 7299057116728623104}]
Jun 20 22:59:26 dist-test-slave-rdl9 krb5kdc[15011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
I20260620 22:59:26.406293 15285 catalog_manager.cc:1347] Generated new cluster ID: 731c4e9c985b432a903e47d0466a3b79
I20260620 22:59:26.406457 15285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:59:26.409004 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.398244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:40069 (local address 127.0.0.1:55014)
0620 22:59:26.398375 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:26.398388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:26.398434 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:26.398828 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:26.398829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:26.399021 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:26.399086 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.399088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.400022 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.400023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.400823 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:26.400826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.401370 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.401371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:26.401428 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.402104 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:26.402109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:26.403905 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:26.407045 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.407047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.407048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.407335 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.407897 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:26.407898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:26.407899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:26.407996 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:26.408604 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:26.408605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:26.408680 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 22:59:26.408737 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:59:26.408884 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 22:59:26.410226 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:26.398518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.1.166.60:40069)
0620 22:59:26.398617 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:26.398619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:26.398640 (+ 21us) server_negotiation.cc:407] Connection header received
0620 22:59:26.398680 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:26.398681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:26.398699 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:26.398810 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:26.399194 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.399895 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.400936 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:26.401240 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:26.401305 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:26.404077 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:26.404079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:26.404079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:26.404090 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:26.406859 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.407527 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.407529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.407530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.407722 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:26.408086 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:26.408087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:26.408119 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:26.408468 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:26.409558 (+ 1090us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:26.409679 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 22:59:26.410036 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:59:26.412536 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6: Bootstrap starting.
I20260620 22:59:26.413720 15167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:26.414109 15167 log.cc:824] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:26.415504 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6: No bootstrap required, opened a new log
I20260620 22:59:26.417917 15167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.418303 15167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:26.418627 15167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80cd152c055e47a38eb40eaf678df6c6, State: Initialized, Role: FOLLOWER
I20260620 22:59:26.419076 15167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } }
I20260620 22:59:26.419703 15292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.419888 15140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80cd152c055e47a38eb40eaf678df6c6"
I20260620 22:59:26.419914 15141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80cd152c055e47a38eb40eaf678df6c6" is_pre_election: true
I20260620 22:59:26.420192 15292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:59:26.420460 15139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:26.421507 15139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [term 1 FOLLOWER]: Refusing update from remote peer ee5b5c4fcc10411c8be95d2f12fefac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:26.421710 15167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:26.421996 15283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:26.423933 15292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee5b5c4fcc10411c8be95d2f12fefac5. Latest consensus state: current_term: 1 leader_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.424170 15292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:59:26.424247 15305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.424306 15305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:59:26.424742 15303 mvcc.cc:205] Tried to move back new op lower bound from 7299057116728623104 to 7299057116707430400. Current Snapshot: MvccSnapshot[applied={T|T < 7299057116728623104}]
W20260620 22:59:26.426999 15306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:59:26.427083 15306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:59:26.428439 15283 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b62a4f5c4c4fe88ced648f6ba87f63" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 33775 } } peers { permanent_uuid: "80cd152c055e47a38eb40eaf678df6c6" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 44131 } } peers { permanent_uuid: "ee5b5c4fcc10411c8be95d2f12fefac5" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 40069 } } }
I20260620 22:59:26.428558 15283 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:26.429965 15285 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:59:26.430822 15285 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:59:26.438271 15285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee5b5c4fcc10411c8be95d2f12fefac5: Generated new TSK 0
I20260620 22:59:26.438567 15285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:59:27.352881 15260 catalog_manager.cc:1259] Loaded cluster ID: 731c4e9c985b432a903e47d0466a3b79
I20260620 22:59:27.352996 15260 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63: loading cluster ID for follower catalog manager: success
I20260620 22:59:27.353782 15260 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63: acquiring CA information for follower catalog manager: success
I20260620 22:59:27.354074 15260 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67b62a4f5c4c4fe88ced648f6ba87f63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:59:27.427718 15306 catalog_manager.cc:1259] Loaded cluster ID: 731c4e9c985b432a903e47d0466a3b79
I20260620 22:59:27.427802 15306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6: loading cluster ID for follower catalog manager: success
I20260620 22:59:27.428865 15306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6: acquiring CA information for follower catalog manager: success
I20260620 22:59:27.429277 15306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80cd152c055e47a38eb40eaf678df6c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:59:27.508590 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15024
I20260620 22:59:27.514081 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15099
I20260620 22:59:27.519241 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15174
2026-06-20T22:59:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2774 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:59:27 dist-test-slave-rdl9 krb5kdc[15317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:59:27 dist-test-slave-rdl9 krb5kdc[15317](info): set up 2 sockets
Jun 20 22:59:27 dist-test-slave-rdl9 krb5kdc[15317](info): commencing operation
krb5kdc: starting...
W20260620 22:59:28.499063 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:59:28 dist-test-slave-rdl9 krb5kdc[15317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:59:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:59:28.625538 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46597
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:46597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:28.717934 15330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:28.718186 15330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:28.718214 15330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:28.719579 15330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:28.719645 15330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:28.719659 15330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:28.719671 15330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:28.719682 15330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:28.721352 15330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:46597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:28.721628 15330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:28.721869 15330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:28.726987 15338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:28.727159 15336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:28.727193 15335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:28.727633 15330 server_base.cc:1034] running on GCE node
I20260620 22:59:28.727874 15330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:28.728295 15330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:28.729620 15330 hybrid_clock.cc:648] HybridClock initialized: now 1781996368729590 us; error 38 us; skew 500 ppm
Jun 20 22:59:28 dist-test-slave-rdl9 krb5kdc[15317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996368, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:28.732625 15330 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:59:28.733142 15330 webserver.cc:460] Webserver started at http://127.1.166.62:46085/ using document root <none> and password file <none>
I20260620 22:59:28.733374 15330 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:28.733424 15330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:28.733560 15330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:28.734726 15330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "aa31a483da4f4582a28d0129f9d75cc7"
format_stamp: "Formatted at 2026-06-20 22:59:28 on dist-test-slave-rdl9"
server_key: "6b452adb21f038869f49c92276637a68"
server_key_iv: "22e0a8385f638e488719c178cda4dedf"
server_key_version: "encryptionkey@0"
I20260620 22:59:28.735080 15330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "aa31a483da4f4582a28d0129f9d75cc7"
format_stamp: "Formatted at 2026-06-20 22:59:28 on dist-test-slave-rdl9"
server_key: "6b452adb21f038869f49c92276637a68"
server_key_iv: "22e0a8385f638e488719c178cda4dedf"
server_key_version: "encryptionkey@0"
I20260620 22:59:28.736465 15330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:59:28.737293 15345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:28.737525 15330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:28.737622 15330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "aa31a483da4f4582a28d0129f9d75cc7"
format_stamp: "Formatted at 2026-06-20 22:59:28 on dist-test-slave-rdl9"
server_key: "6b452adb21f038869f49c92276637a68"
server_key_iv: "22e0a8385f638e488719c178cda4dedf"
server_key_version: "encryptionkey@0"
I20260620 22:59:28.737694 15330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:28.746325 15330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:28.748577 15330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:28.748715 15330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:28.753854 15330 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:46597
I20260620 22:59:28.753906 15397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:46597 every 8 connection(s)
I20260620 22:59:28.754266 15330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:59:28.755164 15398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:28.758067 15398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7: Bootstrap starting.
I20260620 22:59:28.758924 15398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:28.759240 15398 log.cc:824] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:28.760326 15398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7: No bootstrap required, opened a new log
I20260620 22:59:28.761430 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15330
I20260620 22:59:28.761547 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:59:28.761981 1688 external_mini_cluster.cc:1428] Setting key 416f00f10bda12acb563e3085c495042
I20260620 22:59:28.762799 15398 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa31a483da4f4582a28d0129f9d75cc7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46597 } }
I20260620 22:59:28.763042 15398 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:28.763082 15398 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa31a483da4f4582a28d0129f9d75cc7, State: Initialized, Role: FOLLOWER
I20260620 22:59:28.763219 15398 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa31a483da4f4582a28d0129f9d75cc7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46597 } }
I20260620 22:59:28.763306 15398 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:59:28.763341 15398 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:59:28.763391 15398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:28.764619 15398 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa31a483da4f4582a28d0129f9d75cc7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46597 } }
I20260620 22:59:28.764742 15398 leader_election.cc:302] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa31a483da4f4582a28d0129f9d75cc7; no voters:
I20260620 22:59:28.764941 15398 leader_election.cc:288] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:59:28.764982 15403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:28.765094 15403 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [term 1 LEADER]: Becoming Leader. State: Replica: aa31a483da4f4582a28d0129f9d75cc7, State: Running, Role: LEADER
I20260620 22:59:28.765188 15403 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa31a483da4f4582a28d0129f9d75cc7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46597 } }
I20260620 22:59:28.765281 15398 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:28.765553 15403 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa31a483da4f4582a28d0129f9d75cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa31a483da4f4582a28d0129f9d75cc7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46597 } } }
I20260620 22:59:28.765625 15403 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:28.765591 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa31a483da4f4582a28d0129f9d75cc7. Latest consensus state: current_term: 1 leader_uuid: "aa31a483da4f4582a28d0129f9d75cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa31a483da4f4582a28d0129f9d75cc7" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46597 } } }
I20260620 22:59:28.766057 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7 [sys.catalog]: This master's current role is: LEADER
W20260620 22:59:28.767118 15419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:59:28.767184 15419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:59:28.767247 15410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:28.767441 15410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:28.769731 15410 catalog_manager.cc:1347] Generated new cluster ID: 2846fb0ce13e42e38aca4f2327c5a550
I20260620 22:59:28.769814 15410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:59:28 dist-test-slave-rdl9 krb5kdc[15317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:28.776732 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:28.762621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.1.166.62:46597)
0620 22:59:28.762861 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:28.762873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:28.762910 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:59:28.763039 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:28.763049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:28.763116 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:28.763275 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:28.764194 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:28.765530 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:28.766699 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:28.767330 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:28.767454 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:28.771055 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:28.771063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:28.771064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:28.771083 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:28.773488 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:28.774319 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:28.774323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:28.774324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:28.775210 (+ 886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:28.775541 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:28.775542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:28.775542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:28.775950 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:28.776125 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:28.776286 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:59:28.776470 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
I20260620 22:59:28.778477 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:59:28.793431 15410 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:59:28.794227 15410 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:59:28.801129 15410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa31a483da4f4582a28d0129f9d75cc7: Generated new TSK 0
I20260620 22:59:28.801393 15410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:59:28.833851 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15330
2026-06-20T22:59:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1311 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:59:28 dist-test-slave-rdl9 krb5kdc[15425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:59:28 dist-test-slave-rdl9 krb5kdc[15425](info): set up 2 sockets
Jun 20 22:59:28 dist-test-slave-rdl9 krb5kdc[15425](info): commencing operation
krb5kdc: starting...
W20260620 22:59:29.806653 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:59:29 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:59:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:59:29.933003 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34349
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:34349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:30.026016 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:30.026304 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:30.026381 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:30.027900 15438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:30.027984 15438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:30.028005 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:30.028023 15438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:30.028038 15438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:30.030069 15438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:34349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:30.030481 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:30.030751 15438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:30.036298 15443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:30.036535 15438 server_base.cc:1034] running on GCE node
W20260620 22:59:30.036298 15444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:30.036298 15446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:30.036913 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:30.037328 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:30.038506 15438 hybrid_clock.cc:648] HybridClock initialized: now 1781996370038485 us; error 50 us; skew 500 ppm
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:30.042222 15438 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:59:30.043115 15438 webserver.cc:460] Webserver started at http://127.1.166.62:34195/ using document root <none> and password file <none>
I20260620 22:59:30.043448 15438 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:30.043511 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:30.043668 15438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:30.044996 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "ad81541f084140eb8c4c3b539b56376d"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d1d340ea756d84dc844b3c6037c35129"
server_key_iv: "11dc478d8c71cb3a9a75136a8b91a1d7"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.045671 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "ad81541f084140eb8c4c3b539b56376d"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d1d340ea756d84dc844b3c6037c35129"
server_key_iv: "11dc478d8c71cb3a9a75136a8b91a1d7"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.047910 15438 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 22:59:30.049324 15453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.049616 15438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:30.049736 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "ad81541f084140eb8c4c3b539b56376d"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d1d340ea756d84dc844b3c6037c35129"
server_key_iv: "11dc478d8c71cb3a9a75136a8b91a1d7"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.049827 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:30.064743 15438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:30.067633 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:30.067863 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:30.073489 15438 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:34349
I20260620 22:59:30.073590 15505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:34349 every 8 connection(s)
I20260620 22:59:30.073992 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:59:30.075160 15506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:30.077768 15506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Bootstrap starting.
I20260620 22:59:30.078474 15506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:30.078788 15506 log.cc:824] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:30.078958 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15438
I20260620 22:59:30.079059 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:59:30.079264 1688 external_mini_cluster.cc:1428] Setting key fbf96ac05f47aef6ae61164a1de97b03
I20260620 22:59:30.079772 15506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: No bootstrap required, opened a new log
I20260620 22:59:30.081621 15506 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:30.081943 15506 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:30.081990 15506 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad81541f084140eb8c4c3b539b56376d, State: Initialized, Role: FOLLOWER
I20260620 22:59:30.082139 15506 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:30.082237 15506 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:59:30.082273 15506 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:59:30.082320 15506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:30.083503 15506 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:30.083631 15506 leader_election.cc:302] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad81541f084140eb8c4c3b539b56376d; no voters:
I20260620 22:59:30.083840 15506 leader_election.cc:288] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:59:30.083884 15511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:30.084062 15511 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 1 LEADER]: Becoming Leader. State: Replica: ad81541f084140eb8c4c3b539b56376d, State: Running, Role: LEADER
I20260620 22:59:30.084203 15506 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:30.084245 15511 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:30.084605 15511 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad81541f084140eb8c4c3b539b56376d. Latest consensus state: current_term: 1 leader_uuid: "ad81541f084140eb8c4c3b539b56376d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } } }
I20260620 22:59:30.084851 15511 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:30.084779 15512 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad81541f084140eb8c4c3b539b56376d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } } }
I20260620 22:59:30.085120 15512 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:30.085397 15520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:30.085994 15520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:30.088423 15520 catalog_manager.cc:1347] Generated new cluster ID: fd678a15ca8244bcad47e5bcf5a789e6
I20260620 22:59:30.088492 15520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:30.094930 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.079797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.1.166.62:34349)
0620 22:59:30.079965 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.079976 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.080006 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:59:30.081596 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.081606 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.081685 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.081841 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:30.082858 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.084286 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.085683 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.086333 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.086491 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.089740 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:30.089747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:30.089748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:30.089767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:30.092291 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.092942 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.092945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.092946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.093106 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.093356 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.093356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.093357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.094031 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:30.094296 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.094454 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.094666 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260620 22:59:30.107338 15520 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:59:30.108361 15520 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:59:30.134768 15520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Generated new TSK 0
I20260620 22:59:30.134996 15520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:59:30.171933 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34349
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:30.268774 15532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:30.269002 15532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:30.269025 15532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:30.270503 15532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:30.270609 15532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:30.270668 15532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:59:30.273306 15532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:30.273715 15532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:30.274013 15532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:30.279330 15538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:30.279325 15540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:30.279347 15537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:30.279747 15532 server_base.cc:1034] running on GCE node
I20260620 22:59:30.279978 15532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:30.280236 15532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:30.281437 15532 hybrid_clock.cc:648] HybridClock initialized: now 1781996370281397 us; error 66 us; skew 500 ppm
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:30.285070 15532 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:59:30.285614 15532 webserver.cc:460] Webserver started at http://127.1.166.1:40867/ using document root <none> and password file <none>
I20260620 22:59:30.285844 15532 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:30.285897 15532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:30.286029 15532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:30.287259 15532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "ae707534f61743c1bf0ccfcfb97fffb1"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "0a58ce384d18d0ff68296e5622af2e08"
server_key_iv: "4ff2c3dfd598a42b3a8df2d4b975048e"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.287604 15532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "ae707534f61743c1bf0ccfcfb97fffb1"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "0a58ce384d18d0ff68296e5622af2e08"
server_key_iv: "4ff2c3dfd598a42b3a8df2d4b975048e"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.289057 15532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:59:30.289918 15547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.290194 15532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:59:30.290413 15532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "ae707534f61743c1bf0ccfcfb97fffb1"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "0a58ce384d18d0ff68296e5622af2e08"
server_key_iv: "4ff2c3dfd598a42b3a8df2d4b975048e"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.290490 15532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:30.305804 15532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:30.308138 15532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:30.308272 15532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:30.308483 15532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:30.308849 15532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:30.308892 15532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.308923 15532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:30.308941 15532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.315416 15532 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:37577
I20260620 22:59:30.315512 15660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:37577 every 8 connection(s)
I20260620 22:59:30.315934 15532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:59:30.318907 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15532
I20260620 22:59:30.319036 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:59:30.319233 1688 external_mini_cluster.cc:1428] Setting key 2072e4126732fad54203447c08850422
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
I20260620 22:59:30.328320 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.317129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:57143 (local address 127.1.166.62:34349)
0620 22:59:30.317190 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.317192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.317339 (+ 147us) server_negotiation.cc:407] Connection header received
0620 22:59:30.317919 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.317921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.317944 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.318044 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:30.319440 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.320057 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.321200 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.321507 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.321579 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.324851 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:30.324853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:30.324853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:30.324862 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:30.326611 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.327221 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.327222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.327223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.327294 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.327545 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.327546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.327547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.327693 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:30.327754 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.328079 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.328194 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:59:30.328531 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.317026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:57143)
0620 22:59:30.317212 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.317258 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:30.317747 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:30.318134 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:30.318138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:30.318868 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:30.319245 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.319254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.320184 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.320185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.321060 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.321063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.321684 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.321687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.321951 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.322878 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:30.322893 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:30.324713 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:30.326771 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:30.326785 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:30.326789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:30.327132 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:30.327359 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:30.327360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:30.327361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:30.327484 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:30.327763 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:30.327771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:30.327942 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 22:59:30.328048 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 22:59:30.328270 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260620 22:59:30.329409 15661 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34349
I20260620 22:59:30.329552 15661 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:30.329854 15661 heartbeater.cc:511] Master 127.1.166.62:34349 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
I20260620 22:59:30.330610 15470 ts_manager.cc:194] Registered new tserver with Master: ae707534f61743c1bf0ccfcfb97fffb1 (127.1.166.1:37577)
I20260620 22:59:30.331444 15470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:57143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:59:30.345911 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34349
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:30.438666 15666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:30.438915 15666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:30.438943 15666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:30.440264 15666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:30.440326 15666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:30.440358 15666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:59:30.442034 15666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:30.442291 15666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:30.442546 15666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:30.448664 15671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:30.448782 15666 server_base.cc:1034] running on GCE node
W20260620 22:59:30.448664 15672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:30.448664 15674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:30.449185 15666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:30.449524 15666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:30.450768 15666 hybrid_clock.cc:648] HybridClock initialized: now 1781996370450733 us; error 52 us; skew 500 ppm
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:30.453843 15666 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:59:30.454489 15666 webserver.cc:460] Webserver started at http://127.1.166.2:46393/ using document root <none> and password file <none>
I20260620 22:59:30.454725 15666 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:30.454778 15666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:30.454927 15666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:30.456080 15666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "792ea9174d8441a9a68a940c6e4d6fa6"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "f858768b71ad2671c37216fce7ec552c"
server_key_iv: "94b6970635edef96aa688890e656c21b"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.456776 15666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "792ea9174d8441a9a68a940c6e4d6fa6"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "f858768b71ad2671c37216fce7ec552c"
server_key_iv: "94b6970635edef96aa688890e656c21b"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.458683 15666 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 22:59:30.459558 15681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.459757 15666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:30.459846 15666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "792ea9174d8441a9a68a940c6e4d6fa6"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "f858768b71ad2671c37216fce7ec552c"
server_key_iv: "94b6970635edef96aa688890e656c21b"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.459918 15666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:30.477800 15666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:30.480067 15666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:30.480224 15666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:30.480472 15666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:30.480815 15666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:30.480851 15666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.480882 15666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:30.480906 15666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.487321 15666 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:45253
I20260620 22:59:30.487416 15794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:45253 every 8 connection(s)
I20260620 22:59:30.487751 15666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:59:30.492501 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15666
I20260620 22:59:30.492599 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:59:30.492789 1688 external_mini_cluster.cc:1428] Setting key d2725ca15b870c5be9583cd6cdc67f06
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:30.501448 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.488662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:50511 (local address 127.1.166.62:34349)
0620 22:59:30.488759 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.488762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.488959 (+ 197us) server_negotiation.cc:407] Connection header received
0620 22:59:30.489898 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.489901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.489922 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.489984 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:30.491707 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.492319 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.493948 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.494323 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.494439 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.497403 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:30.497406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:30.497406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:30.497418 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:30.499477 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.500057 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.500058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.500059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.500139 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.500607 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.500608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.500608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.500782 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:30.500863 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.501211 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.501312 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 22:59:30.501571 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.488618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:50511)
0620 22:59:30.488824 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.488871 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:30.489671 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:30.490101 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:30.490107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:30.491107 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:30.491500 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.491507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.492545 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.492547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.493844 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.493850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.494497 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.494499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.494651 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.495354 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:30.495367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:30.497258 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:30.499622 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:30.499637 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:30.499640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:30.499965 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:30.500370 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:30.500371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:30.500372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:30.500530 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:30.500885 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:30.500893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:30.501060 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 22:59:30.501174 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 22:59:30.501338 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260620 22:59:30.502244 15795 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34349
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
I20260620 22:59:30.502452 15795 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:30.502727 15795 heartbeater.cc:511] Master 127.1.166.62:34349 requested a full tablet report, sending...
I20260620 22:59:30.503355 15470 ts_manager.cc:194] Registered new tserver with Master: 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253)
I20260620 22:59:30.503799 15470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:50511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:59:30.521220 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34349
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:30.613454 15800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:30.613745 15800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:30.613790 15800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:30.615568 15800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:30.615679 15800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:30.615731 15800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:59:30.617704 15800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:30.618041 15800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:30.618294 15800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:30.623579 15805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:30.623594 15806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:30.623622 15808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:30.623910 15800 server_base.cc:1034] running on GCE node
I20260620 22:59:30.624109 15800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:30.624354 15800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:30.625558 15800 hybrid_clock.cc:648] HybridClock initialized: now 1781996370625539 us; error 35 us; skew 500 ppm
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:30.628775 15800 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:59:30.629264 15800 webserver.cc:460] Webserver started at http://127.1.166.3:33523/ using document root <none> and password file <none>
I20260620 22:59:30.629622 15800 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:30.629832 15800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:30.630053 15800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:30.631613 15800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "2808f675b95649b48adee70db30b9b3f"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d4a57063cb5b28e7b31a80d86db618a1"
server_key_iv: "38e2f032ead4d23f07a7ac2cc03b061a"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.632162 15800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "2808f675b95649b48adee70db30b9b3f"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d4a57063cb5b28e7b31a80d86db618a1"
server_key_iv: "38e2f032ead4d23f07a7ac2cc03b061a"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.633599 15800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:59:30.634428 15815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.634603 15800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:59:30.634680 15800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "2808f675b95649b48adee70db30b9b3f"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d4a57063cb5b28e7b31a80d86db618a1"
server_key_iv: "38e2f032ead4d23f07a7ac2cc03b061a"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.634754 15800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:30.652240 15800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:30.654922 15800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:30.655083 15800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:30.655323 15800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:30.655664 15800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:30.655699 15800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.655728 15800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:30.655746 15800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.661638 15800 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:41575
I20260620 22:59:30.661736 15928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:41575 every 8 connection(s)
I20260620 22:59:30.662135 15800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:59:30.668159 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15800
I20260620 22:59:30.668234 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:59:30.668387 1688 external_mini_cluster.cc:1428] Setting key fe8f5a49e17102cd9930aaf2479c328b
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:30.672071 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.662983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:58287 (local address 127.1.166.62:34349)
0620 22:59:30.663048 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.663050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.663246 (+ 196us) server_negotiation.cc:407] Connection header received
0620 22:59:30.663790 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.663791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.663803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.663868 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:30.664920 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.665437 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.666511 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.666721 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.666784 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.669491 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:30.669492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:30.669492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:30.669500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:30.670773 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.671167 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.671168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.671169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.671220 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.671445 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.671446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.671446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.671567 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:30.671622 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.671897 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.671976 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:59:30.672283 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.662910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:58287)
0620 22:59:30.663078 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.663124 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:30.663622 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:30.663940 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:30.663943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:30.664583 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:30.664803 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.664808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.665560 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.665561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.666432 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.666435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.666792 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.666793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.666892 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.667415 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:30.667424 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:30.669394 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:30.670851 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:30.670858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:30.670861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:30.671099 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:30.671280 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:30.671280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:30.671281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:30.671390 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:30.671625 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:30.671631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:30.671774 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 22:59:30.671867 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 22:59:30.672049 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260620 22:59:30.672791 15929 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34349
I20260620 22:59:30.672883 15929 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:30.673076 15929 heartbeater.cc:511] Master 127.1.166.62:34349 requested a full tablet report, sending...
I20260620 22:59:30.673492 15470 ts_manager.cc:194] Registered new tserver with Master: 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575)
I20260620 22:59:30.673858 15470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:58287
I20260620 22:59:30.681046 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:59:30.688751 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.682426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.1.166.62:34349)
0620 22:59:30.682519 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.682521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.682597 (+ 76us) server_negotiation.cc:407] Connection header received
0620 22:59:30.682668 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.682669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.682679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.682728 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:30.683400 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.683891 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.684631 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.684863 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.684920 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.686310 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:30.686312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:30.686313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:30.686321 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:30.687517 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.687925 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.687926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.687927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.687985 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:30.688152 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:30.688152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:30.688153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:30.688317 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:30.688406 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.688534 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.688622 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 22:59:30.690714 15470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:59:30.692339 15470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:59:30.699939 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.695069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:41575 (local address 127.0.0.1:43238)
0620 22:59:30.695378 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.695424 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:30.695515 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:30.695787 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:30.695790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:30.695792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:30.695880 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.695882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.697080 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.697081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.698257 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.698260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.699556 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.699557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.699661 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.699677 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 22:59:30.699762 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:59:30.699829 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":81,"threads_started":1}
I20260620 22:59:30.699957 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.695260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.1.166.3:41575)
0620 22:59:30.695468 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.695473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.695488 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:59:30.695608 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.695611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.695628 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.695704 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:30.695951 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.696965 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.698371 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.699446 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.699538 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.699619 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.699786 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.699853 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260620 22:59:30.700307 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.695499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.1.166.1:37577)
0620 22:59:30.696153 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.696158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.696170 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:59:30.696213 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.696216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.696232 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.696320 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:30.696542 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.697626 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.699058 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.699707 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.699771 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.699827 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.700170 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.700224 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":192,"threads_started":1}
I20260620 22:59:30.700093 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.695110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:37577 (local address 127.0.0.1:55300)
0620 22:59:30.695480 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.695494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:30.695567 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:30.696402 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:30.696403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:30.696405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:30.696450 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.696451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.697856 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.697857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.698970 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.698972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.699864 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.699865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.699931 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.699934 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:59:30.699978 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:59:30.700023 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":163,"threads_started":1}
I20260620 22:59:30.700299 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.695142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45253 (local address 127.0.0.1:51436)
0620 22:59:30.695643 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.695656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:30.695695 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:30.696223 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:30.696225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:30.696227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:30.696285 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.696286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.697654 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.697655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.698814 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.698817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.700003 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.700005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.700072 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.700078 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:59:30.700110 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 22:59:30.700156 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":139,"threads_started":1}
I20260620 22:59:30.700764 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.695875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.1.166.2:45253)
0620 22:59:30.696067 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.696071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.696081 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:59:30.696133 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.696135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.696146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.696214 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:30.696398 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.697532 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.698926 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.699889 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.700385 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.700462 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.700491 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.700685 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260620 22:59:30.701323 15595 tablet_service.cc:1467] Processing CreateTablet for tablet 58106a150ac74ff9b347297d512c30d4 (DEFAULT_TABLE table=test-table [id=909e79229ff54367ba0202b4fd1d3bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:30.701282 15863 tablet_service.cc:1467] Processing CreateTablet for tablet 58106a150ac74ff9b347297d512c30d4 (DEFAULT_TABLE table=test-table [id=909e79229ff54367ba0202b4fd1d3bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:30.701357 15729 tablet_service.cc:1467] Processing CreateTablet for tablet 58106a150ac74ff9b347297d512c30d4 (DEFAULT_TABLE table=test-table [id=909e79229ff54367ba0202b4fd1d3bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:30.701625 15595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58106a150ac74ff9b347297d512c30d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:30.701645 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58106a150ac74ff9b347297d512c30d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:30.701645 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58106a150ac74ff9b347297d512c30d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:30.704519 15946 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Bootstrap starting.
I20260620 22:59:30.705179 15947 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: Bootstrap starting.
I20260620 22:59:30.705207 15946 tablet_bootstrap.cc:654] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:30.705544 15948 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Bootstrap starting.
I20260620 22:59:30.705839 15947 tablet_bootstrap.cc:654] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:30.705837 15946 log.cc:824] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:30.706229 15948 tablet_bootstrap.cc:654] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:30.706331 15947 log.cc:824] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:30.706838 15946 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: No bootstrap required, opened a new log
I20260620 22:59:30.706838 15948 log.cc:824] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:30.706912 15946 ts_tablet_manager.cc:1397] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:59:30.707279 15947 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: No bootstrap required, opened a new log
I20260620 22:59:30.707351 15947 ts_tablet_manager.cc:1397] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:59:30.707515 15948 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: No bootstrap required, opened a new log
I20260620 22:59:30.707579 15948 ts_tablet_manager.cc:1397] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:59:30.709120 15946 raft_consensus.cc:348] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.709378 15946 raft_consensus.cc:374] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:30.709419 15946 raft_consensus.cc:729] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae707534f61743c1bf0ccfcfb97fffb1, State: Initialized, Role: FOLLOWER
I20260620 22:59:30.709414 15947 raft_consensus.cc:348] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.709563 15947 raft_consensus.cc:374] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:30.709604 15947 raft_consensus.cc:729] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2808f675b95649b48adee70db30b9b3f, State: Initialized, Role: FOLLOWER
I20260620 22:59:30.709561 15946 consensus_queue.cc:260] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.709746 15947 consensus_queue.cc:260] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.709911 15946 ts_tablet_manager.cc:1428] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:59:30.710065 15661 heartbeater.cc:503] Master 127.1.166.62:34349 was elected leader, sending a full tablet report...
I20260620 22:59:30.709879 15948 raft_consensus.cc:348] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.710144 15948 raft_consensus.cc:374] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:30.710184 15948 raft_consensus.cc:729] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792ea9174d8441a9a68a940c6e4d6fa6, State: Initialized, Role: FOLLOWER
I20260620 22:59:30.710368 15948 consensus_queue.cc:260] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.710170 15929 heartbeater.cc:503] Master 127.1.166.62:34349 was elected leader, sending a full tablet report...
I20260620 22:59:30.710016 15947 ts_tablet_manager.cc:1428] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:59:30.710867 15948 ts_tablet_manager.cc:1428] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 22:59:30.711115 15795 heartbeater.cc:503] Master 127.1.166.62:34349 was elected leader, sending a full tablet report...
W20260620 22:59:30.738404 15796 tablet.cc:2369] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:59:30.796312 15953 raft_consensus.cc:482] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:30.796447 15953 raft_consensus.cc:504] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.796852 15953 leader_election.cc:288] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253), ae707534f61743c1bf0ccfcfb97fffb1 (127.1.166.1:37577)
I20260620 22:59:30.800045 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.797161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:37577 (local address 127.1.166.3:56441)
0620 22:59:30.797252 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.797267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:30.797326 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:30.797551 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:30.797553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:30.797561 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:30.797648 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.797650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.798373 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.798374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.799166 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.799169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.799769 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.799770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.799860 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.799864 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:59:30.799916 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:59:30.799955 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 22:59:30.800052 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.797287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:56441 (local address 127.1.166.1:37577)
0620 22:59:30.797334 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.797336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.797351 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:59:30.797387 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.797389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.797401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.797506 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:30.797724 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.798250 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.799275 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.799680 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.799736 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.799746 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.799932 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.799978 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:59:30.800127 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.797286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:50791 (local address 127.1.166.2:45253)
0620 22:59:30.797334 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.797336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.797561 (+ 225us) server_negotiation.cc:407] Connection header received
0620 22:59:30.797602 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.797603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.797619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.797708 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:30.797873 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.798397 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.799362 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.799787 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.799862 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.799872 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.800023 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.800066 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:59:30.800164 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.797161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45253 (local address 127.1.166.3:50791)
0620 22:59:30.797456 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.797470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:30.797541 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:30.797715 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:30.797716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:30.797718 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:30.797783 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.797785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.798482 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.798483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.799261 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:30.799263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.799875 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.799876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:30.799952 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.799956 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:59:30.800008 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:59:30.800045 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":39,"thread_start_us":143,"threads_started":1}
I20260620 22:59:30.800431 15615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58106a150ac74ff9b347297d512c30d4" candidate_uuid: "2808f675b95649b48adee70db30b9b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" is_pre_election: true
I20260620 22:59:30.800529 15749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58106a150ac74ff9b347297d512c30d4" candidate_uuid: "2808f675b95649b48adee70db30b9b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" is_pre_election: true
I20260620 22:59:30.800619 15615 raft_consensus.cc:2454] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2808f675b95649b48adee70db30b9b3f in term 0.
I20260620 22:59:30.800688 15749 raft_consensus.cc:2454] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2808f675b95649b48adee70db30b9b3f in term 0.
I20260620 22:59:30.800872 15819 leader_election.cc:302] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2808f675b95649b48adee70db30b9b3f, ae707534f61743c1bf0ccfcfb97fffb1; no voters:
I20260620 22:59:30.801069 15953 raft_consensus.cc:2781] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:59:30.801112 15953 raft_consensus.cc:482] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:59:30.801126 15953 raft_consensus.cc:3037] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:30.801828 15953 raft_consensus.cc:504] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.801957 15953 leader_election.cc:288] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [CANDIDATE]: Term 1 election: Requested vote from peers 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253), ae707534f61743c1bf0ccfcfb97fffb1 (127.1.166.1:37577)
I20260620 22:59:30.802186 15615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58106a150ac74ff9b347297d512c30d4" candidate_uuid: "2808f675b95649b48adee70db30b9b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae707534f61743c1bf0ccfcfb97fffb1"
I20260620 22:59:30.802265 15615 raft_consensus.cc:3037] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:30.802261 15749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58106a150ac74ff9b347297d512c30d4" candidate_uuid: "2808f675b95649b48adee70db30b9b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792ea9174d8441a9a68a940c6e4d6fa6"
I20260620 22:59:30.802449 15749 raft_consensus.cc:3037] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:30.803360 15615 raft_consensus.cc:2454] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2808f675b95649b48adee70db30b9b3f in term 1.
I20260620 22:59:30.803396 15749 raft_consensus.cc:2454] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2808f675b95649b48adee70db30b9b3f in term 1.
I20260620 22:59:30.803610 15818 leader_election.cc:302] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2808f675b95649b48adee70db30b9b3f, 792ea9174d8441a9a68a940c6e4d6fa6; no voters:
I20260620 22:59:30.803812 15953 raft_consensus.cc:2781] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:30.803989 15953 raft_consensus.cc:686] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 1 LEADER]: Becoming Leader. State: Replica: 2808f675b95649b48adee70db30b9b3f, State: Running, Role: LEADER
I20260620 22:59:30.804100 15953 consensus_queue.cc:237] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:30.805125 15468 catalog_manager.cc:5495] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2808f675b95649b48adee70db30b9b3f (127.1.166.3). New cstate: current_term: 1 leader_uuid: "2808f675b95649b48adee70db30b9b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } health_report { overall_health: HEALTHY } } }
W20260620 22:59:30.817099 15662 tablet.cc:2369] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:59:30.820565 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.817714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.1.166.3:41575)
0620 22:59:30.817771 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:30.817773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:30.817785 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:59:30.817833 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:30.817834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:30.817854 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:30.817933 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:59:30.818180 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.818723 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.819643 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:30.819855 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:30.819919 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:30.820178 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:59:30.820301 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:59:30.820370 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:30.820455 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 22:59:30.820476 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:59:30.822929 15749 raft_consensus.cc:1261] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 1 FOLLOWER]: Refusing update from remote peer 2808f675b95649b48adee70db30b9b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:30.822935 15615 raft_consensus.cc:1261] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Refusing update from remote peer 2808f675b95649b48adee70db30b9b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:30.823447 15956 consensus_queue.cc:1035] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:30.823647 15953 consensus_queue.cc:1035] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:30.826437 15959 mvcc.cc:205] Tried to move back new op lower bound from 7299057134886670336 to 7299057134814408704. Current Snapshot: MvccSnapshot[applied={T|T < 7299057134886670336}]
I20260620 22:59:30.826465 15960 mvcc.cc:205] Tried to move back new op lower bound from 7299057134886670336 to 7299057134814408704. Current Snapshot: MvccSnapshot[applied={T|T < 7299057134886670336}]
I20260620 22:59:30.829779 15470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:59:30.829895 15470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:59:30.830914 15470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Sending DeleteTablet for 3 replicas of tablet 58106a150ac74ff9b347297d512c30d4
I20260620 22:59:30.831372 15863 tablet_service.cc:1514] Processing DeleteTablet for tablet 58106a150ac74ff9b347297d512c30d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:30 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:43238
I20260620 22:59:30.831434 15595 tablet_service.cc:1514] Processing DeleteTablet for tablet 58106a150ac74ff9b347297d512c30d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:30 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:55300
I20260620 22:59:30.831480 15729 tablet_service.cc:1514] Processing DeleteTablet for tablet 58106a150ac74ff9b347297d512c30d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:30 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:51436
I20260620 22:59:30.831714 15970 tablet_replica.cc:330] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: stopping tablet replica
I20260620 22:59:30.831838 15970 raft_consensus.cc:2229] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:59:30.831956 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15532
I20260620 22:59:30.831966 15971 tablet_replica.cc:330] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: stopping tablet replica
I20260620 22:59:30.832046 15970 raft_consensus.cc:2258] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:30.833209 15970 ts_tablet_manager.cc:1905] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:59:30.834406 15971 raft_consensus.cc:2229] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:59:30.834575 15971 raft_consensus.cc:2258] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:30.834576 15970 ts_tablet_manager.cc:1918] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:59:30.834645 15970 log.cc:1196] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/58106a150ac74ff9b347297d512c30d4
I20260620 22:59:30.834962 15970 ts_tablet_manager.cc:1939] T 58106a150ac74ff9b347297d512c30d4 P 2808f675b95649b48adee70db30b9b3f: Deleting consensus metadata
I20260620 22:59:30.835216 15971 ts_tablet_manager.cc:1905] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:59:30.835521 15455 catalog_manager.cc:4841] TS 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575): tablet 58106a150ac74ff9b347297d512c30d4 (table test-table [id=909e79229ff54367ba0202b4fd1d3bcb]) successfully deleted
I20260620 22:59:30.836845 15971 ts_tablet_manager.cc:1918] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:59:30.836967 15971 log.cc:1196] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/58106a150ac74ff9b347297d512c30d4
I20260620 22:59:30.837285 15971 ts_tablet_manager.cc:1939] T 58106a150ac74ff9b347297d512c30d4 P 792ea9174d8441a9a68a940c6e4d6fa6: Deleting consensus metadata
I20260620 22:59:30.837893 15456 catalog_manager.cc:4841] TS 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253): tablet 58106a150ac74ff9b347297d512c30d4 (table test-table [id=909e79229ff54367ba0202b4fd1d3bcb]) successfully deleted
W20260620 22:59:30.839169 15457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:37577 (error 108)
I20260620 22:59:30.839498 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15666
I20260620 22:59:30.840309 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:30.839714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:37577 (local address 127.0.0.1:55316)
0620 22:59:30.840217 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:30.840268 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:37577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475}
W20260620 22:59:30.840888 15457 catalog_manager.cc:4584] TS ae707534f61743c1bf0ccfcfb97fffb1 (127.1.166.1:37577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58106a150ac74ff9b347297d512c30d4: Network error: Client connection negotiation failed: client connection to 127.1.166.1:37577: connect: Connection refused (error 111)
I20260620 22:59:30.845479 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15800
I20260620 22:59:30.851682 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15438
I20260620 22:59:30.857385 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34349
--webserver_interface=127.1.166.62
--webserver_port=34195
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:34349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:30.953064 15973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:30.953421 15973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:30.953459 15973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:30.955938 15973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:30.956055 15973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:30.956074 15973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:30.956089 15973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:30.956113 15973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:30.958621 15973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:34349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=34195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:30.958917 15973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:30.959185 15973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:30.968571 15980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:30.968847 15982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:30.968850 15979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:30.969148 15973 server_base.cc:1034] running on GCE node
I20260620 22:59:30.969412 15973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:30.969719 15973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:30.970898 15973 hybrid_clock.cc:648] HybridClock initialized: now 1781996370970876 us; error 36 us; skew 500 ppm
Jun 20 22:59:30 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996370, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:30.976999 15973 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:59:30.977571 15973 webserver.cc:460] Webserver started at http://127.1.166.62:34195/ using document root <none> and password file <none>
I20260620 22:59:30.977837 15973 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:30.977917 15973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:30.979482 15973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:30.980280 15989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:30.980507 15973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:30.980607 15973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "ad81541f084140eb8c4c3b539b56376d"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d1d340ea756d84dc844b3c6037c35129"
server_key_iv: "11dc478d8c71cb3a9a75136a8b91a1d7"
server_key_version: "encryptionkey@0"
I20260620 22:59:30.981045 15973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:30.990901 15973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:30.993206 15973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:30.993362 15973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:30.997890 15973 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:34349
I20260620 22:59:30.997951 16041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:34349 every 8 connection(s)
I20260620 22:59:30.998478 15973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:59:31.000247 16042 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:59:31.000972 16042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Bootstrap starting.
I20260620 22:59:31.004105 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 15973
I20260620 22:59:31.004441 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:37577
--local_ip_for_outbound_sockets=127.1.166.1
--tserver_master_addrs=127.1.166.62:34349
--webserver_port=40867
--webserver_interface=127.1.166.1
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:59:31.010268 16042 log.cc:824] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:31.013022 16042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:59:31.013371 16042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Bootstrap complete.
I20260620 22:59:31.015676 16042 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:31.015937 16042 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad81541f084140eb8c4c3b539b56376d, State: Initialized, Role: FOLLOWER
I20260620 22:59:31.016089 16042 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:31.016180 16042 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:59:31.016237 16042 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:59:31.016290 16042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:59:31.017422 16042 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:31.017546 16042 leader_election.cc:302] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad81541f084140eb8c4c3b539b56376d; no voters:
I20260620 22:59:31.017724 16042 leader_election.cc:288] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:59:31.017781 16046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:59:31.017905 16046 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 2 LEADER]: Becoming Leader. State: Replica: ad81541f084140eb8c4c3b539b56376d, State: Running, Role: LEADER
I20260620 22:59:31.018008 16042 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:31.018079 16046 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:31.018474 16048 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad81541f084140eb8c4c3b539b56376d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } } }
I20260620 22:59:31.018573 16048 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:31.018522 16046 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad81541f084140eb8c4c3b539b56376d. Latest consensus state: current_term: 2 leader_uuid: "ad81541f084140eb8c4c3b539b56376d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } } }
I20260620 22:59:31.019070 16046 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:31.019838 16062 catalog_manager.cc:1259] Loaded cluster ID: fd678a15ca8244bcad47e5bcf5a789e6
I20260620 22:59:31.019897 16062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: loading cluster ID for follower catalog manager: success
W20260620 22:59:31.020125 16062 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260620 22:59:31.020169 16062 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260620 22:59:31.020215 16055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:31.020602 16055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:31.020680 16055 catalog_manager.cc:1259] Loaded cluster ID: fd678a15ca8244bcad47e5bcf5a789e6
I20260620 22:59:31.020709 16055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:59:31.020859 16055 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 22:59:31.025899 16055 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 22:59:31.028868 16055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:59:31.029152 16055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Loaded TSK: 0
I20260620 22:59:31.029402 16055 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:59:31.095861 16044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:31.096093 16044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:31.096139 16044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:31.097724 16044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:31.097784 16044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:31.097826 16044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:59:31.099699 16044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:37577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=40867
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:31.099961 16044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:31.100215 16044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:31.105388 16070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:31.105388 16069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:31.105551 16044 server_base.cc:1034] running on GCE node
W20260620 22:59:31.105593 16072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:31.105935 16044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:31.106205 16044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:31.107393 16044 hybrid_clock.cc:648] HybridClock initialized: now 1781996371107378 us; error 28 us; skew 500 ppm
Jun 20 22:59:31 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:31.110257 16044 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:59:31.110797 16044 webserver.cc:460] Webserver started at http://127.1.166.1:40867/ using document root <none> and password file <none>
I20260620 22:59:31.111038 16044 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:31.111109 16044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:31.112638 16044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:59:31.113345 16079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.113579 16044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:31.113706 16044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "ae707534f61743c1bf0ccfcfb97fffb1"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "0a58ce384d18d0ff68296e5622af2e08"
server_key_iv: "4ff2c3dfd598a42b3a8df2d4b975048e"
server_key_version: "encryptionkey@0"
I20260620 22:59:31.114238 16044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:31.132010 16044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:31.135080 16044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:31.135264 16044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:31.135558 16044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:31.136240 16086 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:59:31.138132 16044 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:59:31.138216 16044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:59:31.138252 16044 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:59:31.139364 16044 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:59:31.139418 16044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:59:31.139509 16086 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Bootstrap starting.
I20260620 22:59:31.147629 16044 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:37577
I20260620 22:59:31.147936 16193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:37577 every 8 connection(s)
I20260620 22:59:31.148136 16044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:59:31.151132 16086 log.cc:824] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:31.151226 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 16044
I20260620 22:59:31.151510 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:45253
--local_ip_for_outbound_sockets=127.1.166.2
--tserver_master_addrs=127.1.166.62:34349
--webserver_port=46393
--webserver_interface=127.1.166.2
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:59:31.154019 16086 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:59:31.154381 16086 tablet_bootstrap.cc:492] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Bootstrap complete.
I20260620 22:59:31.155316 16086 ts_tablet_manager.cc:1397] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.013s
I20260620 22:59:31.157177 16086 raft_consensus.cc:348] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:31.157518 16086 raft_consensus.cc:729] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae707534f61743c1bf0ccfcfb97fffb1, State: Initialized, Role: FOLLOWER
I20260620 22:59:31.157696 16086 consensus_queue.cc:260] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } }
I20260620 22:59:31.158031 16086 ts_tablet_manager.cc:1428] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
Jun 20 22:59:31 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:31.164589 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.149040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:41657 (local address 127.1.166.62:34349)
0620 22:59:31.150603 (+ 1563us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.150619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.150759 (+ 140us) server_negotiation.cc:407] Connection header received
0620 22:59:31.150966 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.150974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.151040 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.151211 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:31.152177 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.153593 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.155303 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.155934 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.156057 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.159114 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:31.159121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:31.159121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:31.159138 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:31.161973 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:31.162759 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:31.162764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:31.162764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:31.162936 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:31.163319 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:31.163320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:31.163321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:31.163666 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:31.163759 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.164201 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.164372 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":87,"threads_started":1}
I20260620 22:59:31.164613 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.148911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:41657)
0620 22:59:31.149174 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.149223 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.149945 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.151359 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:31.151364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:31.151741 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:31.152024 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.152032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.153833 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.153835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.155175 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.155179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.156080 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.156082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.156276 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.156895 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:31.156913 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:31.158927 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:31.162140 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:31.162158 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:31.162161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:31.162640 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:31.163029 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:31.163030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:31.163030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:31.163217 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:31.163757 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:31.163767 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:31.163976 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 22:59:31.164126 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 22:59:31.164365 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20260620 22:59:31.165879 16194 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34349
I20260620 22:59:31.166049 16194 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:31.166288 16194 heartbeater.cc:511] Master 127.1.166.62:34349 requested a full tablet report, sending...
I20260620 22:59:31.167263 16006 ts_manager.cc:194] Registered new tserver with Master: ae707534f61743c1bf0ccfcfb97fffb1 (127.1.166.1:37577)
I20260620 22:59:31.168583 16006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:41657
I20260620 22:59:31.169751 16194 heartbeater.cc:503] Master 127.1.166.62:34349 was elected leader, sending a full tablet report...
I20260620 22:59:31.175325 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.168012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:37577 (local address 127.0.0.1:55330)
0620 22:59:31.170464 (+ 2452us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.170514 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.170584 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.170897 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:31.170900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:31.170903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:31.170984 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.170986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.172393 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.172395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.173827 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.173832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.174987 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.174989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.175085 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.175100 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:59:31.175191 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 22:59:31.175248 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2411,"thread_start_us":127,"threads_started":1}
I20260620 22:59:31.175559 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.168342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.1.166.1:37577)
0620 22:59:31.170724 (+ 2382us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.170730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.170739 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:59:31.170775 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.170777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.170794 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.171051 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:31.171117 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.172248 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.173964 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.174852 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.174943 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.175017 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.175334 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.175433 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2346,"thread_start_us":77,"threads_started":1}
I20260620 22:59:31.176191 16128 tablet_service.cc:1514] Processing DeleteTablet for tablet 58106a150ac74ff9b347297d512c30d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:30 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:55330
I20260620 22:59:31.176802 16203 tablet_replica.cc:330] stopping tablet replica
I20260620 22:59:31.176920 16203 raft_consensus.cc:2229] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:59:31.176997 16203 raft_consensus.cc:2258] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:31.177330 16203 ts_tablet_manager.cc:1905] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:59:31.179476 16203 ts_tablet_manager.cc:1918] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:59:31.179558 16203 log.cc:1196] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/58106a150ac74ff9b347297d512c30d4
I20260620 22:59:31.179735 16203 ts_tablet_manager.cc:1939] T 58106a150ac74ff9b347297d512c30d4 P ae707534f61743c1bf0ccfcfb97fffb1: Deleting consensus metadata
I20260620 22:59:31.180379 15993 catalog_manager.cc:4841] TS ae707534f61743c1bf0ccfcfb97fffb1 (127.1.166.1:37577): tablet 58106a150ac74ff9b347297d512c30d4 (table test-table [id=909e79229ff54367ba0202b4fd1d3bcb]) successfully deleted
W20260620 22:59:31.249886 16199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:31.250174 16199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:31.250213 16199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:31.252275 16199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:31.252346 16199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:31.252379 16199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:59:31.254228 16199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:45253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=46393
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:31.254536 16199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:31.254791 16199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:31.259912 16208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:31.259982 16211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:31.260151 16199 server_base.cc:1034] running on GCE node
W20260620 22:59:31.260303 16209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:31.260555 16199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:31.260896 16199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:31.262115 16199 hybrid_clock.cc:648] HybridClock initialized: now 1781996371262061 us; error 74 us; skew 500 ppm
Jun 20 22:59:31 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:31.264873 16199 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:59:31.265506 16199 webserver.cc:460] Webserver started at http://127.1.166.2:46393/ using document root <none> and password file <none>
I20260620 22:59:31.265763 16199 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:31.265831 16199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:31.267385 16199 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:31.268111 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.268342 16199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:31.268436 16199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "792ea9174d8441a9a68a940c6e4d6fa6"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "f858768b71ad2671c37216fce7ec552c"
server_key_iv: "94b6970635edef96aa688890e656c21b"
server_key_version: "encryptionkey@0"
I20260620 22:59:31.268810 16199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:31.283188 16199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:31.285560 16199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:31.285722 16199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:31.285950 16199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:31.286278 16199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:31.286315 16199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.286371 16199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:31.286391 16199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.292662 16199 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:45253
I20260620 22:59:31.292716 16331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:45253 every 8 connection(s)
I20260620 22:59:31.293085 16199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:59:31.300302 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 16199
Jun 20 22:59:31 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:31.300567 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:41575
--local_ip_for_outbound_sockets=127.1.166.3
--tserver_master_addrs=127.1.166.62:34349
--webserver_port=33523
--webserver_interface=127.1.166.3
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:59:31.304188 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.294113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:36239 (local address 127.1.166.62:34349)
0620 22:59:31.294167 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.294169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.294431 (+ 262us) server_negotiation.cc:407] Connection header received
0620 22:59:31.294980 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.294982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.295002 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.295080 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:31.296241 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.296766 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.298329 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.298576 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.298649 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.301125 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:31.301128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:31.301128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:31.301137 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:31.302677 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:31.303143 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:31.303144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:31.303145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:31.303207 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:31.303471 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:31.303472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:31.303473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:31.303604 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:31.303664 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.303963 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.304084 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:59:31.304373 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.294050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:36239)
0620 22:59:31.294265 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.294310 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.294843 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.295211 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:31.295215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:31.295856 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:31.296079 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.296084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.296932 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.296935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.298193 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.298200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.298689 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.298691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.298840 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.299396 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:31.299405 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:31.301000 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:31.302775 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:31.302787 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:31.302790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:31.303063 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:31.303267 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:31.303268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:31.303268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:31.303409 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:31.303664 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:31.303672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:31.303832 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 22:59:31.303934 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 22:59:31.304129 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":118,"threads_started":1}
I20260620 22:59:31.304948 16332 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34349
I20260620 22:59:31.305094 16332 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:31.305333 16332 heartbeater.cc:511] Master 127.1.166.62:34349 requested a full tablet report, sending...
I20260620 22:59:31.305788 16006 ts_manager.cc:194] Registered new tserver with Master: 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253)
I20260620 22:59:31.306185 16006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:36239
W20260620 22:59:31.386714 16335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:31.386945 16335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:31.386973 16335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:31.388270 16335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:31.388330 16335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:31.388360 16335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:59:31.390029 16335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:41575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=33523
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:31.390283 16335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:31.390543 16335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:31.395768 16343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:31.395716 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:31.395758 16335 server_base.cc:1034] running on GCE node
W20260620 22:59:31.395716 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:31.396163 16335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:31.396430 16335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:31.397701 16335 hybrid_clock.cc:648] HybridClock initialized: now 1781996371397668 us; error 33 us; skew 500 ppm
Jun 20 22:59:31 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:31.401060 16335 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:59:31.401531 16335 webserver.cc:460] Webserver started at http://127.1.166.3:33523/ using document root <none> and password file <none>
I20260620 22:59:31.401733 16335 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:31.401788 16335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:31.403369 16335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:59:31.404112 16350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.404309 16335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:31.404399 16335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "2808f675b95649b48adee70db30b9b3f"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d4a57063cb5b28e7b31a80d86db618a1"
server_key_iv: "38e2f032ead4d23f07a7ac2cc03b061a"
server_key_version: "encryptionkey@0"
I20260620 22:59:31.404753 16335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:31.416527 16335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:31.419387 16335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:31.419551 16335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:31.419813 16335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:31.420123 16335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:31.420159 16335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.420183 16335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:31.420195 16335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.427522 16335 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:41575
I20260620 22:59:31.427659 16463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:41575 every 8 connection(s)
I20260620 22:59:31.428330 16335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:59:31.436429 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 16335
Jun 20 22:59:31 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:31.441557 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.429334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:55291 (local address 127.1.166.62:34349)
0620 22:59:31.429437 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.429440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.429775 (+ 335us) server_negotiation.cc:407] Connection header received
0620 22:59:31.430542 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.430545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.430561 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.430621 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:31.431790 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.432453 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.433992 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.434230 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.434315 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.437501 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:31.437503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:31.437503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:31.437513 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:31.439269 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:31.439910 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:31.439913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:31.439913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:31.440016 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:31.440503 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:31.440504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:31.440505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:31.440682 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:31.440761 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.441342 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.441452 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 22:59:31.441731 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.429297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:55291)
0620 22:59:31.429499 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.429562 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.430325 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.430698 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:31.430702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:31.431398 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:31.431646 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.431652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.432621 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.432624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.433822 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.433831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.434423 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.434425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.434585 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.435262 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:31.435272 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:31.437391 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:31.439401 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:31.439411 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:31.439414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:31.439808 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:31.440210 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:31.440211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:31.440211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:31.440399 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:31.440814 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:31.440834 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:31.441106 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 22:59:31.441237 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 22:59:31.441427 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260620 22:59:31.442489 16464 heartbeater.cc:347] Connected to a master server at 127.1.166.62:34349
I20260620 22:59:31.442629 16464 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:31.442970 16464 heartbeater.cc:511] Master 127.1.166.62:34349 requested a full tablet report, sending...
I20260620 22:59:31.443579 16006 ts_manager.cc:194] Registered new tserver with Master: 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575)
I20260620 22:59:31.444108 16006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:55291
I20260620 22:59:31.445295 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.437009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.1.166.62:34349)
0620 22:59:31.437202 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.437204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.437218 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:59:31.437290 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.437291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.437306 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.437419 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:31.438224 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.438987 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.439974 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.440275 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.440361 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.442122 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:31.442124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:31.442124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:31.442136 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:31.443819 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:31.444362 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:31.444364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:31.444364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:31.444447 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:31.444786 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:31.444787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:31.444787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:31.444918 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:31.445055 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.445133 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.445218 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260620 22:59:31.446426 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:59:31.449469 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.446938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.1.166.62:34349)
0620 22:59:31.446986 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.446988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.447025 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:59:31.447114 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.447115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.447174 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.447236 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:59:31.447373 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.447912 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.448704 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.448927 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.448985 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.449138 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:59:31.449257 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:59:31.449337 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.449355 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.449383 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:59:31.449977 16006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:59:31.452670 16006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:59:31.455330 16128 tablet_service.cc:1467] Processing CreateTablet for tablet e3f416d3cf6544e0bc763f390be76100 (DEFAULT_TABLE table=test-table [id=66e6e9bcfed84c79a5a9c5359b6fe234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:31.455572 16128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f416d3cf6544e0bc763f390be76100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:31.457047 16086 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Bootstrap starting.
I20260620 22:59:31.457422 16086 tablet_bootstrap.cc:654] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:31.458266 16086 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: No bootstrap required, opened a new log
I20260620 22:59:31.458331 16086 ts_tablet_manager.cc:1397] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:59:31.458495 16086 raft_consensus.cc:348] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.458582 16086 raft_consensus.cc:374] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:31.458642 16086 raft_consensus.cc:729] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae707534f61743c1bf0ccfcfb97fffb1, State: Initialized, Role: FOLLOWER
I20260620 22:59:31.458695 16086 consensus_queue.cc:260] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.458798 16086 ts_tablet_manager.cc:1428] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.459338 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.455004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45253 (local address 127.0.0.1:51440)
0620 22:59:31.455054 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.455065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.455104 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.455532 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:31.455533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:31.455535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:31.455590 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.455591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.456828 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.456830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.457748 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.457751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.458879 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.458881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.458974 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.458981 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 22:59:31.459045 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:59:31.459254 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 22:59:31.459741 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.455115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.1.166.2:45253)
0620 22:59:31.455342 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.455347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.455358 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:59:31.455398 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.455400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.455415 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.455503 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:31.455705 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.456732 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.457838 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.458786 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.458866 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.458951 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.459595 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.459654 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":118,"threads_started":1}
I20260620 22:59:31.460495 16266 tablet_service.cc:1467] Processing CreateTablet for tablet e3f416d3cf6544e0bc763f390be76100 (DEFAULT_TABLE table=test-table [id=66e6e9bcfed84c79a5a9c5359b6fe234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:31.460916 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f416d3cf6544e0bc763f390be76100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:31.461757 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.455026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:41575 (local address 127.0.0.1:43262)
0620 22:59:31.455198 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.455206 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.455240 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.457525 (+ 2285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:31.457527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:31.457529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:31.457577 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.457579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.459112 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.459113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.460273 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.460279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.461514 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.461515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.461587 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.461591 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:59:31.461642 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:59:31.461691 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":134,"threads_started":1}
I20260620 22:59:31.461802 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.457108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.1.166.3:41575)
0620 22:59:31.457338 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.457343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.457355 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:59:31.457399 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.457401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.457419 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.457521 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:31.457640 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.458984 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.460388 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.461321 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.461499 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.461568 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.461668 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.461716 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
I20260620 22:59:31.463019 16398 tablet_service.cc:1467] Processing CreateTablet for tablet e3f416d3cf6544e0bc763f390be76100 (DEFAULT_TABLE table=test-table [id=66e6e9bcfed84c79a5a9c5359b6fe234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:31.463359 16398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f416d3cf6544e0bc763f390be76100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:31.464357 16474 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Bootstrap starting.
I20260620 22:59:31.465116 16474 tablet_bootstrap.cc:654] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:31.465756 16474 log.cc:824] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:31.466732 16474 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: No bootstrap required, opened a new log
I20260620 22:59:31.466816 16474 ts_tablet_manager.cc:1397] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:59:31.467157 16476 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Bootstrap starting.
I20260620 22:59:31.468019 16476 tablet_bootstrap.cc:654] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:31.468705 16476 log.cc:824] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:31.468619 16474 raft_consensus.cc:348] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.468781 16474 raft_consensus.cc:374] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:31.468806 16474 raft_consensus.cc:729] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792ea9174d8441a9a68a940c6e4d6fa6, State: Initialized, Role: FOLLOWER
I20260620 22:59:31.468923 16474 consensus_queue.cc:260] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.469271 16474 ts_tablet_manager.cc:1428] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:59:31.469384 16332 heartbeater.cc:503] Master 127.1.166.62:34349 was elected leader, sending a full tablet report...
I20260620 22:59:31.470109 16476 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: No bootstrap required, opened a new log
I20260620 22:59:31.470227 16476 ts_tablet_manager.cc:1397] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:59:31.472540 16476 raft_consensus.cc:348] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.472738 16476 raft_consensus.cc:374] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:31.472761 16476 raft_consensus.cc:729] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2808f675b95649b48adee70db30b9b3f, State: Initialized, Role: FOLLOWER
I20260620 22:59:31.472862 16476 consensus_queue.cc:260] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.473150 16476 ts_tablet_manager.cc:1428] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:59:31.473222 16464 heartbeater.cc:503] Master 127.1.166.62:34349 was elected leader, sending a full tablet report...
W20260620 22:59:31.544116 16333 tablet.cc:2369] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:59:31.644800 16200 raft_consensus.cc:482] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:31.644961 16200 raft_consensus.cc:504] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.645356 16200 leader_election.cc:288] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253), 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575)
I20260620 22:59:31.649408 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.645574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45253 (local address 127.1.166.1:52421)
0620 22:59:31.645683 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.645757 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.645829 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.646017 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:31.646019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:31.646020 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:31.646086 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.646088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.647272 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.647274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.648246 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.648251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.649055 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.649057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.649156 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.649162 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:59:31.649240 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:59:31.649293 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 22:59:31.649473 16195 tablet.cc:2369] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:59:31.649509 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.645824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:52421 (local address 127.1.166.2:45253)
0620 22:59:31.645895 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.645898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.645905 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:59:31.645935 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.645935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.645949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.646003 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:31.646380 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.647167 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.648359 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.648966 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.649029 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.649043 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.649334 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.649422 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:59:31.649653 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.645709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:60121 (local address 127.1.166.3:41575)
0620 22:59:31.645756 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.645758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.645837 (+ 79us) server_negotiation.cc:407] Connection header received
0620 22:59:31.645943 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.645944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.645957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.646005 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:31.646938 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.647516 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.648755 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.649223 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.649293 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.649305 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.649541 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.649589 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:59:31.649871 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.645574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:41575 (local address 127.1.166.1:60121)
0620 22:59:31.645768 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.645779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.645838 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.646039 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:31.646040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:31.646042 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:31.646775 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.646785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.647630 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.647631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.648537 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:31.648540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.649337 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.649339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:31.649412 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.649417 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:59:31.649725 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 22:59:31.649785 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":70,"thread_start_us":69,"threads_started":1}
I20260620 22:59:31.649865 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f416d3cf6544e0bc763f390be76100" candidate_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" is_pre_election: true
I20260620 22:59:31.650034 16286 raft_consensus.cc:2454] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae707534f61743c1bf0ccfcfb97fffb1 in term 0.
I20260620 22:59:31.650247 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f416d3cf6544e0bc763f390be76100" candidate_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2808f675b95649b48adee70db30b9b3f" is_pre_election: true
I20260620 22:59:31.650302 16082 leader_election.cc:302] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 792ea9174d8441a9a68a940c6e4d6fa6, ae707534f61743c1bf0ccfcfb97fffb1; no voters:
I20260620 22:59:31.650440 16418 raft_consensus.cc:2454] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae707534f61743c1bf0ccfcfb97fffb1 in term 0.
I20260620 22:59:31.650484 16200 raft_consensus.cc:2781] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:59:31.650563 16200 raft_consensus.cc:482] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:59:31.650606 16200 raft_consensus.cc:3037] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:31.651607 16200 raft_consensus.cc:504] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.651769 16200 leader_election.cc:288] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 1 election: Requested vote from peers 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253), 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575)
I20260620 22:59:31.652010 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f416d3cf6544e0bc763f390be76100" candidate_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792ea9174d8441a9a68a940c6e4d6fa6"
I20260620 22:59:31.652029 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f416d3cf6544e0bc763f390be76100" candidate_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2808f675b95649b48adee70db30b9b3f"
I20260620 22:59:31.652105 16418 raft_consensus.cc:3037] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:31.652107 16286 raft_consensus.cc:3037] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:31.653062 16418 raft_consensus.cc:2454] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae707534f61743c1bf0ccfcfb97fffb1 in term 1.
I20260620 22:59:31.653064 16286 raft_consensus.cc:2454] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae707534f61743c1bf0ccfcfb97fffb1 in term 1.
I20260620 22:59:31.653311 16082 leader_election.cc:302] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 792ea9174d8441a9a68a940c6e4d6fa6, ae707534f61743c1bf0ccfcfb97fffb1; no voters:
I20260620 22:59:31.653437 16200 raft_consensus.cc:2781] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:31.653685 16200 raft_consensus.cc:686] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 LEADER]: Becoming Leader. State: Replica: ae707534f61743c1bf0ccfcfb97fffb1, State: Running, Role: LEADER
I20260620 22:59:31.653789 16200 consensus_queue.cc:237] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:31.654904 16005 catalog_manager.cc:5495] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae707534f61743c1bf0ccfcfb97fffb1 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } health_report { overall_health: HEALTHY } } }
W20260620 22:59:31.679150 16465 tablet.cc:2369] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:59:31.693717 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:31.690858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.1.166.1:37577)
0620 22:59:31.691037 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:31.691039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:31.691047 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:59:31.691122 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:31.691123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:31.691153 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:31.691239 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:59:31.691393 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.691933 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.692693 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:31.692889 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:31.692948 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:31.693191 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:59:31.693337 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:59:31.693483 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:31.693575 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 22:59:31.693615 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260620 22:59:31.695442 16286 raft_consensus.cc:1261] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 1 FOLLOWER]: Refusing update from remote peer ae707534f61743c1bf0ccfcfb97fffb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:31.695454 16418 raft_consensus.cc:1261] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 1 FOLLOWER]: Refusing update from remote peer ae707534f61743c1bf0ccfcfb97fffb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:31.695793 16481 consensus_queue.cc:1035] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:31.695976 16200 consensus_queue.cc:1035] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:31.699224 16486 mvcc.cc:205] Tried to move back new op lower bound from 7299057138461532160 to 7299057138294796288. Current Snapshot: MvccSnapshot[applied={T|T < 7299057138461532160}]
I20260620 22:59:31.701447 16006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:59:31.701617 16006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:59:31.702801 16006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Sending DeleteTablet for 3 replicas of tablet e3f416d3cf6544e0bc763f390be76100
I20260620 22:59:31.703388 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f416d3cf6544e0bc763f390be76100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:31 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:51440
I20260620 22:59:31.703474 16398 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f416d3cf6544e0bc763f390be76100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:31 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:43262
I20260620 22:59:31.703676 16128 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f416d3cf6544e0bc763f390be76100 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:31 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:55330
I20260620 22:59:31.703713 16496 tablet_replica.cc:330] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: stopping tablet replica
I20260620 22:59:31.703725 16495 tablet_replica.cc:330] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: stopping tablet replica
I20260620 22:59:31.703851 16495 raft_consensus.cc:2229] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:59:31.703863 16496 raft_consensus.cc:2229] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:59:31.703950 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 16044
I20260620 22:59:31.704000 16495 raft_consensus.cc:2258] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:31.704001 16496 raft_consensus.cc:2258] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:31.704638 16495 ts_tablet_manager.cc:1905] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:59:31.705953 16495 ts_tablet_manager.cc:1918] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:59:31.706022 16495 log.cc:1196] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/e3f416d3cf6544e0bc763f390be76100
I20260620 22:59:31.706259 16495 ts_tablet_manager.cc:1939] T e3f416d3cf6544e0bc763f390be76100 P 792ea9174d8441a9a68a940c6e4d6fa6: Deleting consensus metadata
I20260620 22:59:31.706779 15992 catalog_manager.cc:4841] TS 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253): tablet e3f416d3cf6544e0bc763f390be76100 (table test-table [id=66e6e9bcfed84c79a5a9c5359b6fe234]) successfully deleted
W20260620 22:59:31.708866 15993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:37577 (error 108)
W20260620 22:59:31.710068 16470 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:59:31.709276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:37577 (local address 127.0.0.1:55346)
0620 22:59:31.709331 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:31.709342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:31.709384 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:31.709995 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:37577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:59:31.710162 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 16199
W20260620 22:59:31.710501 15993 catalog_manager.cc:4584] TS ae707534f61743c1bf0ccfcfb97fffb1 (127.1.166.1:37577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3f416d3cf6544e0bc763f390be76100: Network error: Client connection negotiation failed: client connection to 127.1.166.1:37577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:59:31.711472 16496 ts_tablet_manager.cc:1905] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:59:31.713342 16496 ts_tablet_manager.cc:1918] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:59:31.713460 16496 log.cc:1196] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/e3f416d3cf6544e0bc763f390be76100
I20260620 22:59:31.713698 16496 ts_tablet_manager.cc:1939] T e3f416d3cf6544e0bc763f390be76100 P 2808f675b95649b48adee70db30b9b3f: Deleting consensus metadata
I20260620 22:59:31.714188 15991 catalog_manager.cc:4841] TS 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575): tablet e3f416d3cf6544e0bc763f390be76100 (table test-table [id=66e6e9bcfed84c79a5a9c5359b6fe234]) successfully deleted
I20260620 22:59:31.716336 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 16335
I20260620 22:59:31.721668 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 15973
I20260620 22:59:31.727627 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34349
--webserver_interface=127.1.166.62
--webserver_port=34195
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:34349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:31.815426 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:31.815654 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:31.815677 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:31.816880 16499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:31.816929 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:31.816942 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:31.816954 16499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:31.816964 16499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:31.819522 16499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:34349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:34349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=34195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:31.819787 16499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:31.819981 16499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:31.824602 16504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:31.824609 16507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:31.824716 16505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:31.825027 16499 server_base.cc:1034] running on GCE node
I20260620 22:59:31.825259 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:31.825559 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:31.826735 16499 hybrid_clock.cc:648] HybridClock initialized: now 1781996371826719 us; error 34 us; skew 500 ppm
Jun 20 22:59:31 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:31.829336 16499 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:59:31.829845 16499 webserver.cc:460] Webserver started at http://127.1.166.62:34195/ using document root <none> and password file <none>
I20260620 22:59:31.830053 16499 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:31.830114 16499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:31.831815 16499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:31.832888 16514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.833154 16499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:31.833272 16499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "ad81541f084140eb8c4c3b539b56376d"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d1d340ea756d84dc844b3c6037c35129"
server_key_iv: "11dc478d8c71cb3a9a75136a8b91a1d7"
server_key_version: "encryptionkey@0"
I20260620 22:59:31.833689 16499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:31.844545 16499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:31.846949 16499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:31.847100 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:31.851459 16499 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:34349
I20260620 22:59:31.851569 16566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:34349 every 8 connection(s)
I20260620 22:59:31.852015 16499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:59:31.853421 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 16499
I20260620 22:59:31.853866 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:37577
--local_ip_for_outbound_sockets=127.1.166.1
--tserver_master_addrs=127.1.166.62:34349
--webserver_port=40867
--webserver_interface=127.1.166.1
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:59:31.854416 16567 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:59:31.855196 16567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Bootstrap starting.
I20260620 22:59:31.863869 16567 log.cc:824] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:31.867288 16567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:59:31.867769 16567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d: Bootstrap complete.
I20260620 22:59:31.871121 16567 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:31.871474 16567 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad81541f084140eb8c4c3b539b56376d, State: Initialized, Role: FOLLOWER
I20260620 22:59:31.871712 16567 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:31.871816 16567 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:59:31.871865 16567 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:59:31.871917 16567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 2 FOLLOWER]: Advancing to term 3
I20260620 22:59:31.873183 16567 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:31.873304 16567 leader_election.cc:302] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad81541f084140eb8c4c3b539b56376d; no voters:
I20260620 22:59:31.873515 16567 leader_election.cc:288] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 22:59:31.873569 16571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 3 FOLLOWER]: Leader election won for term 3
I20260620 22:59:31.873687 16571 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [term 3 LEADER]: Becoming Leader. State: Replica: ad81541f084140eb8c4c3b539b56376d, State: Running, Role: LEADER
I20260620 22:59:31.873836 16571 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } }
I20260620 22:59:31.873869 16567 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:31.874825 16573 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad81541f084140eb8c4c3b539b56376d. Latest consensus state: current_term: 3 leader_uuid: "ad81541f084140eb8c4c3b539b56376d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } } }
I20260620 22:59:31.874924 16573 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:31.875067 16572 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ad81541f084140eb8c4c3b539b56376d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad81541f084140eb8c4c3b539b56376d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 34349 } } }
I20260620 22:59:31.875138 16572 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad81541f084140eb8c4c3b539b56376d [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:31.875260 16581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:31.876310 16581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:31.876459 16581 catalog_manager.cc:1259] Loaded cluster ID: fd678a15ca8244bcad47e5bcf5a789e6
I20260620 22:59:31.876500 16581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 22:59:31.876722 16581 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1522ecf9ad google::LogMessage::Fail() at ??:0
@ 0x7f1522ed288a google::LogMessage::SendToLog() at ??:0
@ 0x7f1522ecf4de google::LogMessage::Flush() at ??:0
@ 0x7f1522ed0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f152adf00f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f152adf98b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1524d9d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1524d92954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1525bf66db start_thread at ??:0
@ 0x7f15219cd61f clone at ??:0
W20260620 22:59:31.955919 16569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:31.956174 16569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:31.956213 16569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:31.957443 16569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:31.957489 16569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:31.957527 16569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:59:31.959198 16569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:37577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=40867
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:31.959457 16569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:31.959682 16569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:31.964684 16606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:31.964706 16603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:31.964696 16604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:31.965119 16569 server_base.cc:1034] running on GCE node
I20260620 22:59:31.965523 16569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:31.965951 16569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:31.967154 16569 hybrid_clock.cc:648] HybridClock initialized: now 1781996371967103 us; error 71 us; skew 500 ppm
Jun 20 22:59:31 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:31.971354 16569 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:59:31.972081 16569 webserver.cc:460] Webserver started at http://127.1.166.1:40867/ using document root <none> and password file <none>
I20260620 22:59:31.972376 16569 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:31.972434 16569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:31.974332 16569 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:59:31.975272 16613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:31.975518 16569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:31.975631 16569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "ae707534f61743c1bf0ccfcfb97fffb1"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "0a58ce384d18d0ff68296e5622af2e08"
server_key_iv: "4ff2c3dfd598a42b3a8df2d4b975048e"
server_key_version: "encryptionkey@0"
I20260620 22:59:31.976049 16569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:31.988720 16569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:31.991129 16569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:31.991288 16569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:31.991536 16569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:31.992069 16620 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:59:31.994329 16569 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:59:31.994486 16569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260620 22:59:31.994527 16569 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:59:31.995272 16569 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:59:31.995316 16569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:59:31.995431 16620 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Bootstrap starting.
I20260620 22:59:32.003012 16569 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:37577
I20260620 22:59:32.003206 16727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:37577 every 8 connection(s)
I20260620 22:59:32.003502 16569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 22:59:32.008560 16620 log.cc:824] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:32.009856 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 16569
I20260620 22:59:32.010156 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:45253
--local_ip_for_outbound_sockets=127.1.166.2
--tserver_master_addrs=127.1.166.62:34349
--webserver_port=46393
--webserver_interface=127.1.166.2
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:59:32.012691 16620 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:59:32.013041 16620 tablet_bootstrap.cc:492] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Bootstrap complete.
I20260620 22:59:32.014034 16620 ts_tablet_manager.cc:1397] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20260620 22:59:32.015594 16620 raft_consensus.cc:348] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:32.015975 16620 raft_consensus.cc:729] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae707534f61743c1bf0ccfcfb97fffb1, State: Initialized, Role: FOLLOWER
I20260620 22:59:32.016180 16620 consensus_queue.cc:260] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:32.016530 16620 ts_tablet_manager.cc:1428] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
W20260620 22:59:32.101928 16732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:32.102177 16732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:32.102216 16732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:32.103669 16732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:32.103750 16732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:32.103796 16732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:59:32.105479 16732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:45253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=46393
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:32.105762 16732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:32.105986 16732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:32.110937 16738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:32.110919 16739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:32.111187 16741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:32.111277 16732 server_base.cc:1034] running on GCE node
I20260620 22:59:32.111680 16732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:32.111966 16732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:32.113166 16732 hybrid_clock.cc:648] HybridClock initialized: now 1781996372113148 us; error 34 us; skew 500 ppm
Jun 20 22:59:32 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996372, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:32.115998 16732 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:59:32.116566 16732 webserver.cc:460] Webserver started at http://127.1.166.2:46393/ using document root <none> and password file <none>
I20260620 22:59:32.116765 16732 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:32.116804 16732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:32.118165 16732 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:59:32.118846 16748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:32.119014 16732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:59:32.119107 16732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "792ea9174d8441a9a68a940c6e4d6fa6"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "f858768b71ad2671c37216fce7ec552c"
server_key_iv: "94b6970635edef96aa688890e656c21b"
server_key_version: "encryptionkey@0"
I20260620 22:59:32.119474 16732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 22:59:32.119710 16730 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:59:32.004398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:33689)
0620 22:59:32.004644 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.004700 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:32.005474 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:32.119524 (+114050us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":127,"threads_started":1}
W20260620 22:59:32.119907 16728 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:34349 (0 consecutive failures): Network error: Failed to ping master at 127.1.166.62:34349: Client connection negotiation failed: client connection to 127.1.166.62:34349: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:59:32.120237 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:32.120088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:53883)
0620 22:59:32.120153 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.120206 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 22:59:32.120537 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:32.120457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:55325)
0620 22:59:32.120492 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.120517 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 22:59:32.120632 16728 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:59:32.145059 16732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:32.147296 16732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:32.147431 16732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:32.147634 16732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:32.147903 16732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:32.147933 16732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:32.147956 16732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:32.147972 16732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:32.154294 16732 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:45253
I20260620 22:59:32.154392 16861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:45253 every 8 connection(s)
I20260620 22:59:32.154781 16732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:59:32.156196 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 16732
I20260620 22:59:32.156404 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:32.155762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:34061)
0620 22:59:32.156016 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.156167 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":122,"threads_started":1}
I20260620 22:59:32.156509 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:41575
--local_ip_for_outbound_sockets=127.1.166.3
--tserver_master_addrs=127.1.166.62:34349
--webserver_port=33523
--webserver_interface=127.1.166.3
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:32.158505 16862 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:34349 (0 consecutive failures): Network error: Failed to ping master at 127.1.166.62:34349: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
I20260620 22:59:32.158973 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:32.158809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:59273)
0620 22:59:32.158877 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.158930 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 22:59:32.159480 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:32.159373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:49775)
0620 22:59:32.159421 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.159452 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:59:32.159713 16862 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:59:32.247561 16865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:32.247818 16865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:32.247846 16865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:32.249207 16865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:32.249279 16865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:32.249310 16865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:59:32.251101 16865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:41575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=33523
--tserver_master_addrs=127.1.166.62:34349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:32.251405 16865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:32.251658 16865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:32.257296 16870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:32.257306 16871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:32.257345 16873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:32.257501 16865 server_base.cc:1034] running on GCE node
I20260620 22:59:32.257846 16865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:32.258180 16865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:32.259361 16865 hybrid_clock.cc:648] HybridClock initialized: now 1781996372259343 us; error 37 us; skew 500 ppm
Jun 20 22:59:32 dist-test-slave-rdl9 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996372, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:32.262864 16865 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:59:32.263484 16865 webserver.cc:460] Webserver started at http://127.1.166.3:33523/ using document root <none> and password file <none>
I20260620 22:59:32.263738 16865 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:32.263787 16865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:32.265275 16865 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:32.266052 16880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:32.266402 16865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:32.266513 16865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "2808f675b95649b48adee70db30b9b3f"
format_stamp: "Formatted at 2026-06-20 22:59:30 on dist-test-slave-rdl9"
server_key: "d4a57063cb5b28e7b31a80d86db618a1"
server_key_iv: "38e2f032ead4d23f07a7ac2cc03b061a"
server_key_version: "encryptionkey@0"
I20260620 22:59:32.267002 16865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:32.289641 16865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:32.292502 16865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:32.292670 16865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:32.292912 16865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:32.293216 16865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:32.293249 16865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:32.293279 16865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:32.293295 16865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:32.299646 16865 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:41575
I20260620 22:59:32.299739 16993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:41575 every 8 connection(s)
I20260620 22:59:32.300077 16865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 22:59:32.301349 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:32.300781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:40973)
0620 22:59:32.300987 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.301075 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
W20260620 22:59:32.301766 16994 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:34349 (0 consecutive failures): Network error: Failed to ping master at 127.1.166.62:34349: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
I20260620 22:59:32.302057 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:32.301966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:48361)
0620 22:59:32.302008 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.302036 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:59:32.302706 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:32.302531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:39619)
0620 22:59:32.302588 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:32.302679 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 22:59:32.302897 16994 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:59:32.303623 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 16865
I20260620 22:59:33.121348 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:33.121057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:48643)
0620 22:59:33.121232 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:33.121281 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260620 22:59:33.160547 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:33.160238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:36911)
0620 22:59:33.160405 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:33.160464 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260620 22:59:33.203181 17000 raft_consensus.cc:482] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:33.203329 17000 raft_consensus.cc:504] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:33.203661 17000 leader_election.cc:288] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253), 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575)
Jun 20 22:59:33 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.3@KRBTEST.COM
Jun 20 22:59:33 dist-test-slave-rdl9 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996371, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.2@KRBTEST.COM
I20260620 22:59:33.218928 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:33.204160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:45467 (local address 127.1.166.2:45253)
0620 22:59:33.204296 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:33.204311 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:33.204377 (+ 66us) server_negotiation.cc:407] Connection header received
0620 22:59:33.204966 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:33.204969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:33.205023 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:33.205356 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:33.206521 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:33.208006 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:33.210221 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:33.211001 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:33.211150 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:33.215273 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:33.215277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:33.215278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:33.215291 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:33.217180 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:33.217641 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:33.217644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:33.217645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:33.217791 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:33.217991 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:33.217992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:33.217992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:33.218239 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:33.218315 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:33.218657 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 22:59:33.218803 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260620 22:59:33.218930 17002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:33.204001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:46533 (local address 127.1.166.3:41575)
0620 22:59:33.204343 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:33.204350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:33.204373 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:59:33.204939 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:33.204942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:33.205022 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:33.205356 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:33.206504 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:33.208010 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:33.210221 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:33.211001 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:33.211150 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:33.215001 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:33.215008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:33.215009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:33.215028 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:33.216876 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:33.217468 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:33.217472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:33.217472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:33.217608 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:33.217880 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:33.217882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:33.217882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:33.218216 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:33.218300 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:33.218662 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 22:59:33.218803 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":43,"threads_started":1}
I20260620 22:59:33.218925 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:33.203879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45253 (local address 127.1.166.1:45467)
0620 22:59:33.204204 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:33.204222 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:33.204324 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:33.205397 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:33.205401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:33.205964 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:33.206276 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:33.206286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:33.208367 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:33.208368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:33.210057 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:33.210062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:33.211268 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:33.211268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:33.211542 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:33.212537 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:33.212548 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:33.215092 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:33.217299 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:33.217300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:33.217300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:33.217529 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:33.217847 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:33.217848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:33.217848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:33.217921 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:33.218312 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:33.218313 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:33.218476 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 22:59:33.218571 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 22:59:33.218815 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":117,"thread_start_us":103,"threads_started":1}
I20260620 22:59:33.219208 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:33.203871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:41575 (local address 127.1.166.1:46533)
0620 22:59:33.204087 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:33.204102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:33.204175 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:33.205397 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:33.205401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:33.205937 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:33.206276 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:33.206286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:33.208351 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:33.208353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:33.210057 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:33.210062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:33.211263 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:33.211267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:33.211542 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:33.212536 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:33.212548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:33.214784 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:33.217004 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:33.217018 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:33.217021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:33.217363 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:33.217669 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:33.217670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:33.217670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:33.217801 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:33.218292 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:33.218299 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:33.218476 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 22:59:33.218572 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 22:59:33.218907 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":253}
W20260620 22:59:33.220114 16616 leader_election.cc:341] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253): Not found: Tablet not found: e3f416d3cf6544e0bc763f390be76100
W20260620 22:59:33.220202 16615 leader_election.cc:341] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575): Not found: Tablet not found: e3f416d3cf6544e0bc763f390be76100
I20260620 22:59:33.220264 16615 leader_election.cc:302] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae707534f61743c1bf0ccfcfb97fffb1; no voters: 2808f675b95649b48adee70db30b9b3f, 792ea9174d8441a9a68a940c6e4d6fa6
I20260620 22:59:33.220377 17000 raft_consensus.cc:2726] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:59:33.303661 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:33.303367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:56355)
0620 22:59:33.303564 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:33.303614 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260620 22:59:34.122432 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:34.121992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:36631)
0620 22:59:34.122239 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:34.122313 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
I20260620 22:59:34.161337 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:34.161091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:48055)
0620 22:59:34.161217 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:34.161253 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260620 22:59:34.304708 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:34.304405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:47891)
0620 22:59:34.304587 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:34.304639 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260620 22:59:35.105001 17008 raft_consensus.cc:482] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:35.105116 17008 raft_consensus.cc:504] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:35.105263 17008 leader_election.cc:288] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253), 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575)
W20260620 22:59:35.105696 16615 leader_election.cc:341] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575): Not found: Tablet not found: e3f416d3cf6544e0bc763f390be76100
W20260620 22:59:35.105769 16616 leader_election.cc:341] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253): Not found: Tablet not found: e3f416d3cf6544e0bc763f390be76100
I20260620 22:59:35.105847 16616 leader_election.cc:302] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae707534f61743c1bf0ccfcfb97fffb1; no voters: 2808f675b95649b48adee70db30b9b3f, 792ea9174d8441a9a68a940c6e4d6fa6
I20260620 22:59:35.105939 17008 raft_consensus.cc:2726] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:59:35.123543 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:35.123263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:56581)
0620 22:59:35.123425 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:35.123471 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260620 22:59:35.162388 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:35.161992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:38485)
0620 22:59:35.162234 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:35.162300 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":113,"threads_started":1}
I20260620 22:59:35.305749 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:35.305459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:43113)
0620 22:59:35.305644 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:35.305696 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260620 22:59:36.124253 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:36.124001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:39139)
0620 22:59:36.124159 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:36.124201 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260620 22:59:36.163409 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:36.163092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:36327)
0620 22:59:36.163278 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:36.163340 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260620 22:59:36.306695 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:36.306462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:35489)
0620 22:59:36.306600 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:36.306644 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260620 22:59:37.125089 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:37.124835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:45165)
0620 22:59:37.124992 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:37.125039 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260620 22:59:37.164388 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:37.163958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:41643)
0620 22:59:37.164301 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:37.164339 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":174,"threads_started":1}
I20260620 22:59:37.307848 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:37.307356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:57963)
0620 22:59:37.307705 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:37.307754 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":293,"threads_started":1}
I20260620 22:59:38.126242 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:38.125804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:33095)
0620 22:59:38.126022 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:38.126181 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260620 22:59:38.165705 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:38.165255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:60361)
0620 22:59:38.165559 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:38.165621 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
I20260620 22:59:38.308817 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:38.308542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:55251)
0620 22:59:38.308716 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:38.308762 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260620 22:59:38.360302 17021 raft_consensus.cc:482] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:38.360409 17021 raft_consensus.cc:504] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:38.360553 17021 leader_election.cc:288] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253), 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575)
W20260620 22:59:38.360908 16616 leader_election.cc:341] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253): Not found: Tablet not found: e3f416d3cf6544e0bc763f390be76100
W20260620 22:59:38.360981 16615 leader_election.cc:341] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575): Not found: Tablet not found: e3f416d3cf6544e0bc763f390be76100
I20260620 22:59:38.361055 16615 leader_election.cc:302] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae707534f61743c1bf0ccfcfb97fffb1; no voters: 2808f675b95649b48adee70db30b9b3f, 792ea9174d8441a9a68a940c6e4d6fa6
I20260620 22:59:38.361123 17021 raft_consensus.cc:2726] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:59:39.127218 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:39.126934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:42803)
0620 22:59:39.127105 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:39.127150 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
I20260620 22:59:39.166788 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:39.166493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:33793)
0620 22:59:39.166676 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:39.166740 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260620 22:59:39.309679 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:39.309420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:58415)
0620 22:59:39.309581 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:39.309631 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260620 22:59:40.128230 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:40.127906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:42247)
0620 22:59:40.128113 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:40.128177 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
I20260620 22:59:40.167977 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:40.167643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:51465)
0620 22:59:40.167849 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:40.167916 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
I20260620 22:59:40.310708 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:40.310328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:44987)
0620 22:59:40.310584 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:40.310654 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":87,"threads_started":1}
I20260620 22:59:41.129243 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:41.128894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:46559)
0620 22:59:41.129117 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:41.129189 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260620 22:59:41.168936 17029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:41.168626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:41577)
0620 22:59:41.168805 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:41.168866 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260620 22:59:41.311726 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:41.311473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:52563)
0620 22:59:41.311631 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:41.311672 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260620 22:59:42.130133 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:42.129773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:54827)
0620 22:59:42.130008 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:42.130064 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":141,"threads_started":1}
I20260620 22:59:42.170068 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:42.169656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:60347)
0620 22:59:42.169921 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:42.169999 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":76,"threads_started":1}
I20260620 22:59:42.312668 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:42.312344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:58179)
0620 22:59:42.312570 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:42.312613 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260620 22:59:43.131215 17034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:43.130911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:46235)
0620 22:59:43.131107 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:43.131162 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260620 22:59:43.171128 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:43.170745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:53701)
0620 22:59:43.170968 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:43.171038 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":118,"threads_started":1}
I20260620 22:59:43.236872 17036 raft_consensus.cc:482] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:43.236994 17036 raft_consensus.cc:504] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792ea9174d8441a9a68a940c6e4d6fa6" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45253 } } peers { permanent_uuid: "2808f675b95649b48adee70db30b9b3f" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41575 } } peers { permanent_uuid: "ae707534f61743c1bf0ccfcfb97fffb1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 37577 } }
I20260620 22:59:43.237170 17036 leader_election.cc:288] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253), 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575)
W20260620 22:59:43.237586 16616 leader_election.cc:341] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 792ea9174d8441a9a68a940c6e4d6fa6 (127.1.166.2:45253): Not found: Tablet not found: e3f416d3cf6544e0bc763f390be76100
W20260620 22:59:43.237675 16615 leader_election.cc:341] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2808f675b95649b48adee70db30b9b3f (127.1.166.3:41575): Not found: Tablet not found: e3f416d3cf6544e0bc763f390be76100
I20260620 22:59:43.237716 16615 leader_election.cc:302] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae707534f61743c1bf0ccfcfb97fffb1; no voters: 2808f675b95649b48adee70db30b9b3f, 792ea9174d8441a9a68a940c6e4d6fa6
I20260620 22:59:43.237780 17036 raft_consensus.cc:2726] T e3f416d3cf6544e0bc763f390be76100 P ae707534f61743c1bf0ccfcfb97fffb1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:59:43.313625 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:43.313337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:36049)
0620 22:59:43.313520 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:43.313569 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260620 22:59:44.132140 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:44.131897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:55957)
0620 22:59:44.132051 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:44.132090 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260620 22:59:44.171989 17039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:44.171655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:36575)
0620 22:59:44.171801 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:44.171940 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260620 22:59:44.314610 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:44.314249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:37069)
0620 22:59:44.314500 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:44.314554 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":135,"threads_started":1}
I20260620 22:59:45.132968 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:45.132725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:53725)
0620 22:59:45.132870 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:45.132918 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260620 22:59:45.172909 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:45.172598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:59789)
0620 22:59:45.172787 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:45.172860 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260620 22:59:45.315533 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:45.315263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:58125)
0620 22:59:45.315421 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:45.315479 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260620 22:59:46.133875 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:46.133597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:36675)
0620 22:59:46.133776 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:46.133825 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260620 22:59:46.173764 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:46.173514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:44147)
0620 22:59:46.173670 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:46.173716 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260620 22:59:46.316540 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:46.316222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:46819)
0620 22:59:46.316439 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:46.316487 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260620 22:59:47.135072 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:47.134671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.1:55041)
0620 22:59:47.134939 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:47.135016 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260620 22:59:47.174525 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:47.174293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.2:44253)
0620 22:59:47.174435 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:47.174479 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260620 22:59:47.307549 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 16569
I20260620 22:59:47.313575 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 16732
I20260620 22:59:47.317476 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:47.317224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:34349 (local address 127.1.166.3:43339)
0620 22:59:47.317367 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:47.317420 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:34349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260620 22:59:47.319741 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 16865
2026-06-20T22:59:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18487 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:59:47 dist-test-slave-rdl9 krb5kdc[17052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:59:47 dist-test-slave-rdl9 krb5kdc[17052](info): set up 2 sockets
Jun 20 22:59:47 dist-test-slave-rdl9 krb5kdc[17052](info): commencing operation
krb5kdc: starting...
W20260620 22:59:48.306751 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 22:59:48 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 22:59:48.434806 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45921
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:45921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:48.527117 17065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:48.527395 17065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:48.527437 17065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:48.529313 17065 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:59:48.529390 17065 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:59:48.529408 17065 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:59:48.529426 17065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:48.529448 17065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:48.529469 17065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:48.529495 17065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:48.529518 17065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:48.532166 17065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:45921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:48.532714 17065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:48.533053 17065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:48.539103 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:48.539153 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:48.539170 17065 server_base.cc:1034] running on GCE node
W20260620 22:59:48.539116 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:48.539655 17065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:48.539975 17065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:48.541148 17065 hybrid_clock.cc:648] HybridClock initialized: now 1781996388541129 us; error 31 us; skew 500 ppm
Jun 20 22:59:48 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:48.544102 17065 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 22:59:48.544780 17065 webserver.cc:460] Webserver started at http://127.1.166.62:37141/ using document root <none> and password file <none>
I20260620 22:59:48.545060 17065 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:48.545117 17065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:48.545286 17065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:48.546630 17065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "dc6990539d3a4f70979d534e8a5a933d"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "894ff1a4506225367fc5daba0688475f"
server_key_iv: "bf3de0da63f7967abab9505a664377f4"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.547061 17065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "dc6990539d3a4f70979d534e8a5a933d"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "894ff1a4506225367fc5daba0688475f"
server_key_iv: "bf3de0da63f7967abab9505a664377f4"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.548775 17065 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 22:59:48.550122 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:48.550441 17065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:48.550565 17065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "dc6990539d3a4f70979d534e8a5a933d"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "894ff1a4506225367fc5daba0688475f"
server_key_iv: "bf3de0da63f7967abab9505a664377f4"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.550665 17065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:48.572538 17065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:48.575029 17065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:48.575191 17065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:48.579394 17065 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:45921
I20260620 22:59:48.579440 17132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:45921 every 8 connection(s)
I20260620 22:59:48.579828 17065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 22:59:48.580476 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 17065
I20260620 22:59:48.580614 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 22:59:48.580852 1688 external_mini_cluster.cc:1428] Setting key a365db8e7a480f1c55eff0902ca26d75
I20260620 22:59:48.580853 17133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:48.584096 17133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Bootstrap starting.
I20260620 22:59:48.585147 17133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:48.585498 17133 log.cc:824] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:48.586269 17133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: No bootstrap required, opened a new log
Jun 20 22:59:48 dist-test-slave-rdl9 krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:48.589540 17133 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 22:59:48.589700 17133 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:48.589732 17133 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6990539d3a4f70979d534e8a5a933d, State: Initialized, Role: FOLLOWER
I20260620 22:59:48.589865 17133 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 22:59:48.589941 17133 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:59:48.589972 17133 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:59:48.590032 17133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:48.590709 17133 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 22:59:48.590816 17133 leader_election.cc:302] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6990539d3a4f70979d534e8a5a933d; no voters:
I20260620 22:59:48.590997 17133 leader_election.cc:288] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:59:48.591034 17138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:48.591166 17138 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 1 LEADER]: Becoming Leader. State: Replica: dc6990539d3a4f70979d534e8a5a933d, State: Running, Role: LEADER
I20260620 22:59:48.591262 17138 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 22:59:48.591320 17133 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:48.591617 17138 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6990539d3a4f70979d534e8a5a933d. Latest consensus state: current_term: 1 leader_uuid: "dc6990539d3a4f70979d534e8a5a933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } } }
I20260620 22:59:48.591861 17139 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc6990539d3a4f70979d534e8a5a933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } } }
I20260620 22:59:48.592013 17139 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:48.592471 17147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:48.591814 17138 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:48.592995 17147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:48.595353 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:48.581463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.1.166.62:45921)
0620 22:59:48.581640 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:48.581648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:48.581675 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:59:48.582300 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:48.582307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:48.582390 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:48.582533 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:48.583349 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.584635 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.585958 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.586490 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.586582 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:48.590005 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:48.590013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:48.590014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:48.590034 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:48.592476 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:48.593894 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:48.593899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:48.593899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:48.594024 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:48.594274 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:48.594275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:48.594275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:48.594641 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:48.594785 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:48.594986 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 22:59:48.595128 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260620 22:59:48.595484 17147 catalog_manager.cc:1347] Generated new cluster ID: 4630d6da8f2e4420bf08823e5520baf7
I20260620 22:59:48.595517 17147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:59:48.621856 17147 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:59:48.622575 17147 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:59:48.626178 17147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Generated new TSK 0
I20260620 22:59:48.626475 17147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 22:59:48.653388 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:45921
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:48.745704 17159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:48.745954 17159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:48.745985 17159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:48.747372 17159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:48.747444 17159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:48.747475 17159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 22:59:48.749131 17159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:48.749388 17159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:48.749652 17159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:48.754732 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:48.754751 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:48.755038 17159 server_base.cc:1034] running on GCE node
W20260620 22:59:48.754757 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:48.755281 17159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:48.755518 17159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:48.756698 17159 hybrid_clock.cc:648] HybridClock initialized: now 1781996388756647 us; error 64 us; skew 500 ppm
Jun 20 22:59:48 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:48.759523 17159 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 22:59:48.760044 17159 webserver.cc:460] Webserver started at http://127.1.166.1:46859/ using document root <none> and password file <none>
I20260620 22:59:48.760255 17159 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:48.760304 17159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:48.760433 17159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:48.761489 17159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "484d79d194e20508c2450699f41f0c74"
server_key_iv: "550d7ca30d3a98a3a6ade859e8b1e954"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.761814 17159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "484d79d194e20508c2450699f41f0c74"
server_key_iv: "550d7ca30d3a98a3a6ade859e8b1e954"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.763206 17159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:59:48.764008 17174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:48.764200 17159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:59:48.764287 17159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "484d79d194e20508c2450699f41f0c74"
server_key_iv: "550d7ca30d3a98a3a6ade859e8b1e954"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.764351 17159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:48.772712 17159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:48.775053 17159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:48.775219 17159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:48.775449 17159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:48.775763 17159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:48.775799 17159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:48.775830 17159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:48.775854 17159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:48.781881 17159 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:36801
I20260620 22:59:48.782006 17287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:36801 every 8 connection(s)
I20260620 22:59:48.782284 17159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:59:48 dist-test-slave-rdl9 krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:48.789773 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 17159
I20260620 22:59:48.789906 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 22:59:48.790159 1688 external_mini_cluster.cc:1428] Setting key 626753fbbec82f22e86f2cb3de35265e
I20260620 22:59:48.793771 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:48.783206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:41325 (local address 127.1.166.62:45921)
0620 22:59:48.783369 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:48.783371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:48.783389 (+ 18us) server_negotiation.cc:407] Connection header received
0620 22:59:48.783951 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:48.783952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:48.783968 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:48.784063 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:48.785069 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.785726 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.786938 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.787277 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.787354 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:48.789766 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:48.789768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:48.789768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:48.789775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:48.791640 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:48.792373 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:48.792374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:48.792375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:48.792463 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:48.792738 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:48.792739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:48.792740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:48.792944 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:48.793009 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:48.793566 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 22:59:48.793675 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260620 22:59:48.793958 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:48.783098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:41325)
0620 22:59:48.783274 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:48.783320 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:48.783805 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:48.784125 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:48.784129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:48.784729 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:48.784940 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:48.784945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.785834 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.785835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:48.786810 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:48.786814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.787394 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.787396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:48.787529 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:48.788115 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:48.788125 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:48.789637 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:48.791764 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:48.791776 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:48.791779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:48.792256 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:48.792533 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:48.792534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:48.792535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:48.792674 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:48.793007 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:48.793017 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:48.793349 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 22:59:48.793502 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 22:59:48.793714 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260620 22:59:48.794428 17288 heartbeater.cc:347] Connected to a master server at 127.1.166.62:45921
I20260620 22:59:48.794538 17288 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:48.794732 17288 heartbeater.cc:511] Master 127.1.166.62:45921 requested a full tablet report, sending...
I20260620 22:59:48.795271 17097 ts_manager.cc:194] Registered new tserver with Master: 41f1e86860dc4fa2a626b0f4b3f2ba80 (127.1.166.1:36801)
I20260620 22:59:48.796216 17097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:41325
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 22:59:48.817798 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:45921
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:48.907877 17293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:48.908186 17293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:48.908372 17293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:48.909982 17293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:48.910061 17293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:48.910097 17293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 22:59:48.912667 17293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:48.913007 17293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:48.913275 17293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:48.918663 17301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:48.918727 17299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:48.918865 17298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:48.918747 17293 server_base.cc:1034] running on GCE node
I20260620 22:59:48.919116 17293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:48.919381 17293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:48.920574 17293 hybrid_clock.cc:648] HybridClock initialized: now 1781996388920546 us; error 38 us; skew 500 ppm
Jun 20 22:59:48 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:48.923485 17293 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 22:59:48.924088 17293 webserver.cc:460] Webserver started at http://127.1.166.2:36811/ using document root <none> and password file <none>
I20260620 22:59:48.924315 17293 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:48.924363 17293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:48.924459 17293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:48.925673 17293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "3c5d366ef0884ddc65df5c9b807c2acd"
server_key_iv: "5b6286dc9b0c94ab9584f4f2e5c4c7bd"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.926051 17293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "3c5d366ef0884ddc65df5c9b807c2acd"
server_key_iv: "5b6286dc9b0c94ab9584f4f2e5c4c7bd"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.927515 17293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:59:48.928462 17308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:48.928864 17293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:59:48.928987 17293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "3c5d366ef0884ddc65df5c9b807c2acd"
server_key_iv: "5b6286dc9b0c94ab9584f4f2e5c4c7bd"
server_key_version: "encryptionkey@0"
I20260620 22:59:48.929112 17293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:48.946266 17293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:48.949424 17293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:48.949580 17293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:48.949908 17293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:48.950275 17293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:48.950310 17293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:48.950374 17293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:48.950392 17293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:48.957389 17293 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:40103
I20260620 22:59:48.957446 17421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:40103 every 8 connection(s)
I20260620 22:59:48.957785 17293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 22:59:48.963603 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 17293
I20260620 22:59:48.963738 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 22:59:48.963948 1688 external_mini_cluster.cc:1428] Setting key 16771c44daa267f64ff576b1aa5600e7
Jun 20 22:59:48 dist-test-slave-rdl9 krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996388, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:48.969103 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:48.958541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:49911 (local address 127.1.166.62:45921)
0620 22:59:48.958611 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:48.958613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:48.958819 (+ 206us) server_negotiation.cc:407] Connection header received
0620 22:59:48.959308 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:48.959309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:48.959321 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:48.959465 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:48.960438 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.960939 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.961984 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.962184 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.962260 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:48.965296 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:48.965297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:48.965298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:48.965312 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:48.967058 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:48.967875 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:48.967877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:48.967877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:48.967959 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:48.968286 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:48.968287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:48.968287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:48.968411 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:48.968467 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:48.968881 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 22:59:48.969010 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:59:48.969272 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:48.958490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:49911)
0620 22:59:48.958651 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:48.958689 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:48.959182 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:48.959537 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:48.959541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:48.960152 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:48.960354 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:48.960359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.961051 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.961052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:48.961880 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:48.961882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:48.962260 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:48.962261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:48.962404 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:48.962950 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:48.962963 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:48.965124 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:48.967208 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:48.967224 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:48.967226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:48.967650 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:48.968042 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:48.968043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:48.968043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:48.968197 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:48.968477 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:48.968483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:48.968678 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 22:59:48.968799 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 22:59:48.969001 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260620 22:59:48.969851 17422 heartbeater.cc:347] Connected to a master server at 127.1.166.62:45921
I20260620 22:59:48.969969 17422 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:48.970170 17422 heartbeater.cc:511] Master 127.1.166.62:45921 requested a full tablet report, sending...
I20260620 22:59:48.970636 17097 ts_manager.cc:194] Registered new tserver with Master: a574ffe02c2c4e1fa1a099e965bbd0ea (127.1.166.2:40103)
I20260620 22:59:48.971499 17097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:49911
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 22:59:48.991564 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:45921
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:59:49.082141 17427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:49.082423 17427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:49.082464 17427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:49.083966 17427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:59:49.084036 17427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:49.084082 17427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 22:59:49.085932 17427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:59:49.086248 17427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:49.086555 17427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:49.091741 17432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:49.091741 17435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:49.091828 17433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:49.091974 17427 server_base.cc:1034] running on GCE node
I20260620 22:59:49.092245 17427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:49.092501 17427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:49.093740 17427 hybrid_clock.cc:648] HybridClock initialized: now 1781996389093712 us; error 42 us; skew 500 ppm
Jun 20 22:59:49 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:59:49.096516 17427 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 22:59:49.097020 17427 webserver.cc:460] Webserver started at http://127.1.166.3:40015/ using document root <none> and password file <none>
I20260620 22:59:49.097208 17427 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:49.097247 17427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:49.097342 17427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:49.098418 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "38c8f5e7b4a64d419d0bf956b139dc83"
format_stamp: "Formatted at 2026-06-20 22:59:49 on dist-test-slave-rdl9"
server_key: "10cdf0204ac3a21b5f5df233792d3040"
server_key_iv: "665b86dab2e925d814274d1028445d43"
server_key_version: "encryptionkey@0"
I20260620 22:59:49.098771 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "38c8f5e7b4a64d419d0bf956b139dc83"
format_stamp: "Formatted at 2026-06-20 22:59:49 on dist-test-slave-rdl9"
server_key: "10cdf0204ac3a21b5f5df233792d3040"
server_key_iv: "665b86dab2e925d814274d1028445d43"
server_key_version: "encryptionkey@0"
I20260620 22:59:49.100127 17427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:59:49.100900 17442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:49.101060 17427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:59:49.101146 17427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "38c8f5e7b4a64d419d0bf956b139dc83"
format_stamp: "Formatted at 2026-06-20 22:59:49 on dist-test-slave-rdl9"
server_key: "10cdf0204ac3a21b5f5df233792d3040"
server_key_iv: "665b86dab2e925d814274d1028445d43"
server_key_version: "encryptionkey@0"
I20260620 22:59:49.101217 17427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:49.129834 17427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:49.132648 17427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:49.132803 17427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:49.133038 17427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:49.133376 17427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:49.133409 17427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:49.133440 17427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:49.133463 17427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:49.139509 17427 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:35583
I20260620 22:59:49.139575 17555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:35583 every 8 connection(s)
I20260620 22:59:49.139989 17427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:59:49 dist-test-slave-rdl9 krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996389, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 22:59:49.147512 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 17427
I20260620 22:59:49.147606 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 22:59:49.147931 1688 external_mini_cluster.cc:1428] Setting key 3ae7da0a60e988317577d81953071a6a
I20260620 22:59:49.151808 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.140773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:49723 (local address 127.1.166.62:45921)
0620 22:59:49.140839 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:49.140841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:49.141040 (+ 199us) server_negotiation.cc:407] Connection header received
0620 22:59:49.141565 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:49.141566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:49.141580 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:49.141641 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:49.142758 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.143276 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.144652 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.144895 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.144965 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.148099 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:49.148101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:49.148101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:49.148111 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:49.149972 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:49.150529 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:49.150530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:49.150531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:49.150591 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:49.151005 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:49.151006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:49.151007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:49.151168 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:49.151235 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:49.151573 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 22:59:49.151686 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:59:49.152032 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.140721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:49723)
0620 22:59:49.140906 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:49.140951 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:49.141465 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:49.141728 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:49.141732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:49.142402 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:59:49.142644 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.142656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.143376 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.143377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.144476 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.144483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.144984 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.144985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.145130 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.145732 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:59:49.145743 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:59:49.147945 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:59:49.150099 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:49.150111 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:49.150113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:49.150448 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:49.150646 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:59:49.150647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:59:49.150647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:59:49.150875 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:59:49.151236 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:59:49.151245 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:59:49.151416 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 22:59:49.151521 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 22:59:49.151736 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260620 22:59:49.152670 17556 heartbeater.cc:347] Connected to a master server at 127.1.166.62:45921
I20260620 22:59:49.152799 17556 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:49.153020 17556 heartbeater.cc:511] Master 127.1.166.62:45921 requested a full tablet report, sending...
I20260620 22:59:49.153425 17097 ts_manager.cc:194] Registered new tserver with Master: 38c8f5e7b4a64d419d0bf956b139dc83 (127.1.166.3:35583)
I20260620 22:59:49.153800 17097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:49723
I20260620 22:59:49.160593 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:59:49.171042 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.162052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.1.166.62:45921)
0620 22:59:49.162116 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:49.162118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:49.162130 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:59:49.162241 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:49.162242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:49.162252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:49.162314 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:59:49.163010 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.163509 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.164960 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.165992 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.166087 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.167713 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:59:49.167715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:59:49.167715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:59:49.167725 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:59:49.169499 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:49.170182 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:49.170183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:49.170184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:49.170244 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:59:49.170459 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:59:49.170459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:59:49.170460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:59:49.170623 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:59:49.170715 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:49.170861 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 22:59:49.170933 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:59:49.172678 17097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:59:49.174314 17097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:59:49.181540 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.177159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:36801 (local address 127.0.0.1:56264)
0620 22:59:49.177453 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:49.177463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:49.177526 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:49.177765 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:49.177766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:49.177768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:49.177848 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.177850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.179241 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.179242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.180411 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.180413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.181073 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.181074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.181280 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.181299 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 22:59:49.181374 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:59:49.181441 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":37,"thread_start_us":66,"threads_started":1}
I20260620 22:59:49.181563 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.177080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.1.166.3:35583)
0620 22:59:49.177290 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:49.177296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:49.177378 (+ 82us) server_negotiation.cc:407] Connection header received
0620 22:59:49.177530 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:49.177533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:49.177549 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:49.177638 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:49.177935 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.179132 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.180279 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.181034 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.181118 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.181177 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:49.181412 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 22:59:49.181466 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20260620 22:59:49.181625 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.177523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.1.166.2:40103)
0620 22:59:49.177687 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:49.177692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:49.177867 (+ 175us) server_negotiation.cc:407] Connection header received
0620 22:59:49.177910 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:49.177912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:49.177927 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:49.177993 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:49.178182 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.178926 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.180431 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.181288 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.181357 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.181410 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:49.181499 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 22:59:49.181543 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260620 22:59:49.181799 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.176933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:35583 (local address 127.0.0.1:51212)
0620 22:59:49.177277 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:49.177321 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:49.177396 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:49.177736 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:49.177739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:49.177755 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:49.177856 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.177858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.179260 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.179260 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.180193 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.180196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.181214 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.181215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.181282 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.181305 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 22:59:49.181693 (+ 388us) client_negotiation.cc:240] Negotiation successful
0620 22:59:49.181745 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":29,"thread_start_us":80,"threads_started":1}
I20260620 22:59:49.181883 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.177282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.1.166.1:36801)
0620 22:59:49.177448 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:49.177451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:49.177580 (+ 129us) server_negotiation.cc:407] Connection header received
0620 22:59:49.177621 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:49.177623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:49.177637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:49.177699 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:49.177923 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.179133 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.180479 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.180991 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.181057 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.181122 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:49.181737 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 22:59:49.181798 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260620 22:59:49.182654 17490 tablet_service.cc:1467] Processing CreateTablet for tablet f10d1f784b1941e1bad7639c9cfe3f5e (DEFAULT_TABLE table=test-table [id=10c1992fc57246f28c5dffdd5748b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:49.182739 17222 tablet_service.cc:1467] Processing CreateTablet for tablet f10d1f784b1941e1bad7639c9cfe3f5e (DEFAULT_TABLE table=test-table [id=10c1992fc57246f28c5dffdd5748b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:49.183060 17222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10d1f784b1941e1bad7639c9cfe3f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:49.183060 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10d1f784b1941e1bad7639c9cfe3f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:49.186447 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.177308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:40103 (local address 127.0.0.1:37586)
0620 22:59:49.177755 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:49.177766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:49.177817 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:49.178064 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:49.178065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:49.178067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:49.178114 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.178115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.179029 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.179030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.180330 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.180334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.181355 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.181356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.181426 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.181430 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:59:49.181474 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:59:49.181512 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":61,"threads_started":1}
I20260620 22:59:49.186987 17573 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: Bootstrap starting.
I20260620 22:59:49.187638 17573 tablet_bootstrap.cc:654] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:49.187611 17356 tablet_service.cc:1467] Processing CreateTablet for tablet f10d1f784b1941e1bad7639c9cfe3f5e (DEFAULT_TABLE table=test-table [id=10c1992fc57246f28c5dffdd5748b832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:49.187938 17356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10d1f784b1941e1bad7639c9cfe3f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:49.188175 17573 log.cc:824] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:49.188526 17574 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Bootstrap starting.
I20260620 22:59:49.189201 17574 tablet_bootstrap.cc:654] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:49.189689 17574 log.cc:824] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:49.190294 17573 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: No bootstrap required, opened a new log
I20260620 22:59:49.190408 17573 ts_tablet_manager.cc:1397] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 22:59:49.190513 17574 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: No bootstrap required, opened a new log
I20260620 22:59:49.190601 17574 ts_tablet_manager.cc:1397] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:59:49.191159 17577 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Bootstrap starting.
I20260620 22:59:49.191916 17577 tablet_bootstrap.cc:654] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:49.192577 17577 log.cc:824] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:49.192643 17573 raft_consensus.cc:348] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.192873 17573 raft_consensus.cc:374] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:49.192919 17573 raft_consensus.cc:729] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c8f5e7b4a64d419d0bf956b139dc83, State: Initialized, Role: FOLLOWER
I20260620 22:59:49.193049 17573 consensus_queue.cc:260] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.192966 17574 raft_consensus.cc:348] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.193212 17574 raft_consensus.cc:374] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:49.193265 17574 raft_consensus.cc:729] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f1e86860dc4fa2a626b0f4b3f2ba80, State: Initialized, Role: FOLLOWER
I20260620 22:59:49.193347 17573 ts_tablet_manager.cc:1428] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:59:49.193404 17574 consensus_queue.cc:260] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.193598 17556 heartbeater.cc:503] Master 127.1.166.62:45921 was elected leader, sending a full tablet report...
I20260620 22:59:49.193758 17574 ts_tablet_manager.cc:1428] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:59:49.193948 17288 heartbeater.cc:503] Master 127.1.166.62:45921 was elected leader, sending a full tablet report...
I20260620 22:59:49.194634 17577 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: No bootstrap required, opened a new log
I20260620 22:59:49.194710 17577 ts_tablet_manager.cc:1397] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 22:59:49.196370 17577 raft_consensus.cc:348] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.196545 17577 raft_consensus.cc:374] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:49.196579 17577 raft_consensus.cc:729] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a574ffe02c2c4e1fa1a099e965bbd0ea, State: Initialized, Role: FOLLOWER
I20260620 22:59:49.196702 17577 consensus_queue.cc:260] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.196998 17577 ts_tablet_manager.cc:1428] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:59:49.197034 17422 heartbeater.cc:503] Master 127.1.166.62:45921 was elected leader, sending a full tablet report...
I20260620 22:59:49.206611 17579 raft_consensus.cc:482] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:59:49.206851 17579 raft_consensus.cc:504] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.207425 17579 leader_election.cc:288] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41f1e86860dc4fa2a626b0f4b3f2ba80 (127.1.166.1:36801), a574ffe02c2c4e1fa1a099e965bbd0ea (127.1.166.2:40103)
W20260620 22:59:49.208491 17423 tablet.cc:2369] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:59:49.211150 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.207835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:45569 (local address 127.1.166.2:40103)
0620 22:59:49.207920 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:49.207922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:49.207993 (+ 71us) server_negotiation.cc:407] Connection header received
0620 22:59:49.208063 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:49.208063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:49.208075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:49.208127 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:49.208372 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.209078 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.210301 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.210790 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.210866 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.210880 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:49.211015 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:59:49.211060 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:59:49.211172 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.207666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:40103 (local address 127.1.166.3:45569)
0620 22:59:49.207921 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:49.207931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:49.208002 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:49.208141 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:49.208142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:49.208143 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:49.208264 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.208267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.209204 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.209205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.210154 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.210158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.210879 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.210880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.210951 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.210955 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:59:49.211003 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:59:49.211058 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":2,"thread_start_us":203,"threads_started":1}
I20260620 22:59:49.211599 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.207556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:36801 (local address 127.1.166.3:45969)
0620 22:59:49.207611 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:49.207624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:59:49.207682 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:59:49.207979 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:59:49.207979 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:59:49.207982 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:59:49.208063 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.208065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.209113 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.209115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.210258 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:59:49.210262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.211321 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.211322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:59:49.211425 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.211429 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:59:49.211478 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:59:49.211532 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":69}
I20260620 22:59:49.211618 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.207650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:45969 (local address 127.1.166.1:36801)
0620 22:59:49.207708 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:49.207711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:49.207719 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:59:49.207838 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:49.207840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:49.207858 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:49.207960 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:59:49.208142 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.208994 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.210412 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.211182 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.211270 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.211285 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:49.211494 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 22:59:49.211551 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:59:49.211609 17376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10d1f784b1941e1bad7639c9cfe3f5e" candidate_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" is_pre_election: true
I20260620 22:59:49.211796 17376 raft_consensus.cc:2454] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c8f5e7b4a64d419d0bf956b139dc83 in term 0.
I20260620 22:59:49.212241 17443 leader_election.cc:302] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c8f5e7b4a64d419d0bf956b139dc83, a574ffe02c2c4e1fa1a099e965bbd0ea; no voters:
I20260620 22:59:49.212317 17242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10d1f784b1941e1bad7639c9cfe3f5e" candidate_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" is_pre_election: true
I20260620 22:59:49.212498 17242 raft_consensus.cc:2454] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c8f5e7b4a64d419d0bf956b139dc83 in term 0.
I20260620 22:59:49.212482 17579 raft_consensus.cc:2781] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:59:49.212563 17579 raft_consensus.cc:482] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:59:49.212600 17579 raft_consensus.cc:3037] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:49.213562 17579 raft_consensus.cc:504] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.213817 17579 leader_election.cc:288] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [CANDIDATE]: Term 1 election: Requested vote from peers 41f1e86860dc4fa2a626b0f4b3f2ba80 (127.1.166.1:36801), a574ffe02c2c4e1fa1a099e965bbd0ea (127.1.166.2:40103)
I20260620 22:59:49.214088 17242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10d1f784b1941e1bad7639c9cfe3f5e" candidate_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80"
I20260620 22:59:49.214179 17242 raft_consensus.cc:3037] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:49.214136 17376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10d1f784b1941e1bad7639c9cfe3f5e" candidate_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea"
I20260620 22:59:49.214249 17376 raft_consensus.cc:3037] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:49.214999 17242 raft_consensus.cc:2454] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c8f5e7b4a64d419d0bf956b139dc83 in term 1.
I20260620 22:59:49.215062 17376 raft_consensus.cc:2454] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c8f5e7b4a64d419d0bf956b139dc83 in term 1.
I20260620 22:59:49.215206 17443 leader_election.cc:302] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c8f5e7b4a64d419d0bf956b139dc83, a574ffe02c2c4e1fa1a099e965bbd0ea; no voters:
I20260620 22:59:49.215306 17579 raft_consensus.cc:2781] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:49.215490 17579 raft_consensus.cc:686] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 LEADER]: Becoming Leader. State: Replica: 38c8f5e7b4a64d419d0bf956b139dc83, State: Running, Role: LEADER
I20260620 22:59:49.215596 17579 consensus_queue.cc:237] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 22:59:49.216522 17097 catalog_manager.cc:5495] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38c8f5e7b4a64d419d0bf956b139dc83 (127.1.166.3). New cstate: current_term: 1 leader_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } health_report { overall_health: UNKNOWN } } }
I20260620 22:59:49.229328 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.226507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.1.166.3:35583)
0620 22:59:49.226562 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:59:49.226564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:59:49.226571 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:59:49.226629 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:59:49.226630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:59:49.226653 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:59:49.226716 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:59:49.227026 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.227526 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.228359 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:59:49.228585 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:59:49.228658 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:59:49.228870 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:59:49.229010 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:59:49.229083 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:59:49.229189 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 22:59:49.229230 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:59:49.231637 17242 raft_consensus.cc:1261] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Refusing update from remote peer 38c8f5e7b4a64d419d0bf956b139dc83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:49.231727 17376 raft_consensus.cc:1261] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Refusing update from remote peer 38c8f5e7b4a64d419d0bf956b139dc83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:49.232141 17579 consensus_queue.cc:1035] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:49.232354 17583 consensus_queue.cc:1035] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:59:49.234937 17586 mvcc.cc:205] Tried to move back new op lower bound from 7299057210289958912 to 7299057210227961856. Current Snapshot: MvccSnapshot[applied={T|T < 7299057210289958912}]
I20260620 22:59:49.235985 17587 mvcc.cc:205] Tried to move back new op lower bound from 7299057210289958912 to 7299057210227961856. Current Snapshot: MvccSnapshot[applied={T|T < 7299057210289958912 or (T in {7299057210289958912})}]
I20260620 22:59:49.236497 17588 mvcc.cc:205] Tried to move back new op lower bound from 7299057210289958912 to 7299057210227961856. Current Snapshot: MvccSnapshot[applied={T|T < 7299057210289958912}]
I20260620 22:59:49.237682 17097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:59:49.237795 17097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:59:49.238574 17097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Sending DeleteTablet for 3 replicas of tablet f10d1f784b1941e1bad7639c9cfe3f5e
I20260620 22:59:49.238885 17490 tablet_service.cc:1514] Processing DeleteTablet for tablet f10d1f784b1941e1bad7639c9cfe3f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:49 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:51212
I20260620 22:59:49.239049 17356 tablet_service.cc:1514] Processing DeleteTablet for tablet f10d1f784b1941e1bad7639c9cfe3f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:49 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:37586
I20260620 22:59:49.239162 17595 tablet_replica.cc:330] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: stopping tablet replica
I20260620 22:59:49.239208 17222 tablet_service.cc:1514] Processing DeleteTablet for tablet f10d1f784b1941e1bad7639c9cfe3f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:49 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:56264
I20260620 22:59:49.239264 17595 raft_consensus.cc:2229] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:59:49.239362 17595 raft_consensus.cc:2258] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:49.239535 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 17159
I20260620 22:59:49.239784 17595 ts_tablet_manager.cc:1905] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:59:49.241603 17595 ts_tablet_manager.cc:1918] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:59:49.241683 17595 log.cc:1196] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/f10d1f784b1941e1bad7639c9cfe3f5e
I20260620 22:59:49.241930 17595 ts_tablet_manager.cc:1939] T f10d1f784b1941e1bad7639c9cfe3f5e P 38c8f5e7b4a64d419d0bf956b139dc83: Deleting consensus metadata
I20260620 22:59:49.242401 17081 catalog_manager.cc:4841] TS 38c8f5e7b4a64d419d0bf956b139dc83 (127.1.166.3:35583): tablet f10d1f784b1941e1bad7639c9cfe3f5e (table test-table [id=10c1992fc57246f28c5dffdd5748b832]) successfully deleted
I20260620 22:59:49.246557 17596 tablet_replica.cc:330] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: stopping tablet replica
I20260620 22:59:49.246737 17596 raft_consensus.cc:2229] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:59:49.246878 17596 raft_consensus.cc:2258] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:59:49.247364 17596 ts_tablet_manager.cc:1905] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 22:59:49.247535 17081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:36801 (error 108)
I20260620 22:59:49.247660 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 17293
I20260620 22:59:49.248471 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.248099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:36801 (local address 127.0.0.1:56268)
0620 22:59:49.248402 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:49.248437 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:36801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
W20260620 22:59:49.249006 17081 catalog_manager.cc:4584] TS 41f1e86860dc4fa2a626b0f4b3f2ba80 (127.1.166.1:36801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f10d1f784b1941e1bad7639c9cfe3f5e: Network error: Client connection negotiation failed: client connection to 127.1.166.1:36801: connect: Connection refused (error 111)
I20260620 22:59:49.254092 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 17427
I20260620 22:59:49.254251 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:59:49.254113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:40103 (local address 127.0.0.1:37594)
0620 22:59:49.254173 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 22:59:49.254205 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.2:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:59:49.258939 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 17065
I20260620 23:00:09.265250 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45921
--webserver_interface=127.1.166.62
--webserver_port=37141
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:45921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:09.356871 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:09.357195 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:09.357239 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:09.359014 17599 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:00:09.359061 17599 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:00:09.359079 17599 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:00:09.359095 17599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:09.359112 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:09.359131 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:09.359154 17599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:09.359174 17599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:09.361058 17599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:45921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:09.361444 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:09.361754 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:09.371994 17606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:09.371982 17605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:09.371982 17608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:09.372151 17599 server_base.cc:1034] running on GCE node
I20260620 23:00:09.372493 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:09.372875 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:09.374105 17599 hybrid_clock.cc:648] HybridClock initialized: now 1781996409374051 us; error 66 us; skew 500 ppm
Jun 20 23:00:09 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:09.377321 17599 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:00:09.377846 17599 webserver.cc:460] Webserver started at http://127.1.166.62:37141/ using document root <none> and password file <none>
I20260620 23:00:09.378098 17599 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:09.378155 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:09.379753 17599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:09.380726 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:09.380954 17599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:09.381059 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "dc6990539d3a4f70979d534e8a5a933d"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "894ff1a4506225367fc5daba0688475f"
server_key_iv: "bf3de0da63f7967abab9505a664377f4"
server_key_version: "encryptionkey@0"
I20260620 23:00:09.381510 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:09.393894 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:09.396443 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:09.396597 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:09.400985 17599 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:45921
I20260620 23:00:09.401037 17667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:45921 every 8 connection(s)
I20260620 23:00:09.401464 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:09.402037 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 17599
I20260620 23:00:09.402426 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:36801
--local_ip_for_outbound_sockets=127.1.166.1
--tserver_master_addrs=127.1.166.62:45921
--webserver_port=46859
--webserver_interface=127.1.166.1
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:00:09.403286 17668 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:00:09.404124 17668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Bootstrap starting.
I20260620 23:00:09.416827 17668 log.cc:824] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:09.419747 17668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:00:09.420087 17668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Bootstrap complete.
I20260620 23:00:09.423429 17668 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 23:00:09.423791 17668 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6990539d3a4f70979d534e8a5a933d, State: Initialized, Role: FOLLOWER
I20260620 23:00:09.423970 17668 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 23:00:09.424046 17668 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:09.424089 17668 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:09.424129 17668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:00:09.425518 17668 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 23:00:09.425638 17668 leader_election.cc:302] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6990539d3a4f70979d534e8a5a933d; no voters:
I20260620 23:00:09.425804 17668 leader_election.cc:288] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:00:09.425935 17672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:00:09.426059 17668 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:09.426098 17672 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 2 LEADER]: Becoming Leader. State: Replica: dc6990539d3a4f70979d534e8a5a933d, State: Running, Role: LEADER
I20260620 23:00:09.426215 17672 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 23:00:09.426504 17673 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc6990539d3a4f70979d534e8a5a933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } } }
I20260620 23:00:09.426584 17673 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:09.426888 17674 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6990539d3a4f70979d534e8a5a933d. Latest consensus state: current_term: 2 leader_uuid: "dc6990539d3a4f70979d534e8a5a933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } } }
I20260620 23:00:09.427110 17674 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:09.427902 17688 catalog_manager.cc:1259] Loaded cluster ID: 4630d6da8f2e4420bf08823e5520baf7
I20260620 23:00:09.427950 17688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: loading cluster ID for follower catalog manager: success
I20260620 23:00:09.429296 17688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: acquiring CA information for follower catalog manager: success
I20260620 23:00:09.429883 17688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:00:09.429994 17680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:09.430596 17680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:09.430701 17680 catalog_manager.cc:1259] Loaded cluster ID: 4630d6da8f2e4420bf08823e5520baf7
I20260620 23:00:09.430737 17680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:09.430943 17680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:09.431033 17680 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Loaded TSK: 0
I20260620 23:00:09.449815 17680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Generated new TSK 1
I20260620 23:00:09.449996 17680 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:00:09.501883 17670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:09.502107 17670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:09.502142 17670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:09.503427 17670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:09.503471 17670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:09.503512 17670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 23:00:09.505410 17670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:36801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=46859
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:09.505702 17670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:09.505923 17670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:09.511124 17698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:09.511143 17695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:09.511124 17696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:09.511893 17670 server_base.cc:1034] running on GCE node
I20260620 23:00:09.512218 17670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:09.512814 17670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:09.514075 17670 hybrid_clock.cc:648] HybridClock initialized: now 1781996409514051 us; error 48 us; skew 500 ppm
Jun 20 23:00:09 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:09.517431 17670 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 23:00:09.518127 17670 webserver.cc:460] Webserver started at http://127.1.166.1:46859/ using document root <none> and password file <none>
I20260620 23:00:09.518409 17670 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:09.518483 17670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:09.519968 17670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:09.520742 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:09.520956 17670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:09.521060 17670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "484d79d194e20508c2450699f41f0c74"
server_key_iv: "550d7ca30d3a98a3a6ade859e8b1e954"
server_key_version: "encryptionkey@0"
I20260620 23:00:09.521456 17670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:09.535965 17670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:09.538457 17670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:09.538607 17670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:09.538836 17670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:09.539350 17712 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:00:09.541428 17670 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:00:09.541597 17670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:00:09.541683 17670 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:00:09.542765 17670 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:00:09.542806 17670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:09.542953 17712 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Bootstrap starting.
I20260620 23:00:09.550817 17670 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:36801
I20260620 23:00:09.551224 17670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 23:00:09.554667 17819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:36801 every 8 connection(s)
I20260620 23:00:09.555131 17712 log.cc:824] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:09.557454 17712 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:00:09.557725 17712 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Bootstrap complete.
I20260620 23:00:09.558840 17712 ts_tablet_manager.cc:1397] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20260620 23:00:09.559579 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 17670
I20260620 23:00:09.559886 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:40103
--local_ip_for_outbound_sockets=127.1.166.2
--tserver_master_addrs=127.1.166.62:45921
--webserver_port=36811
--webserver_interface=127.1.166.2
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:00:09.560300 17712 raft_consensus.cc:348] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 23:00:09.560626 17712 raft_consensus.cc:729] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f1e86860dc4fa2a626b0f4b3f2ba80, State: Initialized, Role: FOLLOWER
I20260620 23:00:09.560756 17712 consensus_queue.cc:260] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 23:00:09.561482 17712 ts_tablet_manager.cc:1428] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
Jun 20 23:00:09 dist-test-slave-rdl9 krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:09.567840 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.552133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:54559 (local address 127.1.166.62:45921)
0620 23:00:09.555005 (+ 2872us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.555014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.555046 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:00:09.555168 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.555173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.555229 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.555396 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:09.556205 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.557626 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.559187 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.559903 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.560290 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.563955 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:09.563963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:09.563964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:09.563980 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:09.565926 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.566503 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.566507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.566508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.566611 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.566860 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.566862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.566862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.567104 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:09.567180 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.567483 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.567613 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2819,"thread_start_us":90,"threads_started":1}
I20260620 23:00:09.567828 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.551993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:54559)
0620 23:00:09.552217 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.552269 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:09.552770 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:09.555442 (+ 2672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:09.555447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:09.555795 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:09.556059 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.556067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.557789 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.557790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.559082 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.559086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.560395 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.560397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.560548 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.561810 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:09.561825 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:09.563751 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:09.566078 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:09.566090 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:09.566092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:09.566419 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:09.566678 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:09.566679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:09.566679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:09.566791 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:09.567173 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:09.567179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:09.567322 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 23:00:09.567405 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:00:09.567603 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
I20260620 23:00:09.568538 17820 heartbeater.cc:347] Connected to a master server at 127.1.166.62:45921
I20260620 23:00:09.568642 17820 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:09.568820 17820 heartbeater.cc:511] Master 127.1.166.62:45921 requested a full tablet report, sending...
I20260620 23:00:09.569525 17632 ts_manager.cc:194] Registered new tserver with Master: 41f1e86860dc4fa2a626b0f4b3f2ba80 (127.1.166.1:36801)
I20260620 23:00:09.570845 17632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:54559
I20260620 23:00:09.571719 17820 heartbeater.cc:503] Master 127.1.166.62:45921 was elected leader, sending a full tablet report...
I20260620 23:00:09.577745 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.570456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.1.166.1:36801)
0620 23:00:09.570708 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.570713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.574779 (+ 4066us) server_negotiation.cc:407] Connection header received
0620 23:00:09.574823 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.574826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.574840 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.574904 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:09.575110 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.575824 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.576828 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.577332 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.577399 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.577461 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.577614 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.577659 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260620 23:00:09.577745 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.570133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:36801 (local address 127.0.0.1:45614)
0620 23:00:09.574529 (+ 4396us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.574582 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:09.574689 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:09.574920 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:09.574923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:09.574926 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:09.575041 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.575043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.575909 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.575910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.576737 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.576740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.577412 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.577413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.577511 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.577526 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:00:09.577592 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:00:09.577655 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4355,"thread_start_us":119,"threads_started":1}
I20260620 23:00:09.578409 17754 tablet_service.cc:1514] Processing DeleteTablet for tablet f10d1f784b1941e1bad7639c9cfe3f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:49 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:45614
I20260620 23:00:09.578941 17829 tablet_replica.cc:330] stopping tablet replica
I20260620 23:00:09.579047 17829 raft_consensus.cc:2229] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:09.579119 17829 raft_consensus.cc:2258] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:09.579434 17829 ts_tablet_manager.cc:1905] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:09.582186 17829 ts_tablet_manager.cc:1918] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:09.582274 17829 log.cc:1196] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/f10d1f784b1941e1bad7639c9cfe3f5e
I20260620 23:00:09.582541 17829 ts_tablet_manager.cc:1939] T f10d1f784b1941e1bad7639c9cfe3f5e P 41f1e86860dc4fa2a626b0f4b3f2ba80: Deleting consensus metadata
I20260620 23:00:09.583245 17616 catalog_manager.cc:4841] TS 41f1e86860dc4fa2a626b0f4b3f2ba80 (127.1.166.1:36801): tablet f10d1f784b1941e1bad7639c9cfe3f5e (table test-table [id=10c1992fc57246f28c5dffdd5748b832]) successfully deleted
W20260620 23:00:09.664740 17825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:09.664968 17825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:09.664994 17825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:09.666249 17825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:09.666304 17825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:09.666363 17825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 23:00:09.667899 17825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:40103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=36811
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:09.668143 17825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:09.668393 17825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:09.673481 17835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:09.673483 17834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:09.673549 17837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:09.673569 17825 server_base.cc:1034] running on GCE node
I20260620 23:00:09.673914 17825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:09.674150 17825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:09.675343 17825 hybrid_clock.cc:648] HybridClock initialized: now 1781996409675323 us; error 33 us; skew 500 ppm
Jun 20 23:00:09 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:09.678217 17825 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 23:00:09.678783 17825 webserver.cc:460] Webserver started at http://127.1.166.2:36811/ using document root <none> and password file <none>
I20260620 23:00:09.678995 17825 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:09.679064 17825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:09.680498 17825 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:09.681336 17844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:09.681664 17825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:09.681845 17825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "3c5d366ef0884ddc65df5c9b807c2acd"
server_key_iv: "5b6286dc9b0c94ab9584f4f2e5c4c7bd"
server_key_version: "encryptionkey@0"
I20260620 23:00:09.682437 17825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:09.692737 17825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:09.695637 17825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:09.695794 17825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:09.696012 17825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:09.696468 17851 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:00:09.697940 17825 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:00:09.698000 17825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:00:09.698040 17825 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:00:09.698784 17825 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:00:09.698824 17825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:00:09.698925 17851 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Bootstrap starting.
I20260620 23:00:09.700459 17851 log.cc:824] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:09.702950 17851 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:00:09.703233 17851 tablet_bootstrap.cc:492] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Bootstrap complete.
I20260620 23:00:09.703399 17851 ts_tablet_manager.cc:1397] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:00:09.704802 17851 raft_consensus.cc:348] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 23:00:09.705134 17851 raft_consensus.cc:729] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a574ffe02c2c4e1fa1a099e965bbd0ea, State: Initialized, Role: FOLLOWER
I20260620 23:00:09.705413 17851 consensus_queue.cc:260] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } }
I20260620 23:00:09.705770 17825 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:40103
I20260620 23:00:09.706121 17825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 23:00:09.706037 17851 ts_tablet_manager.cc:1428] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:09.706049 17959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:40103 every 8 connection(s)
I20260620 23:00:09.707010 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 17825
I20260620 23:00:09.707314 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:35583
--local_ip_for_outbound_sockets=127.1.166.3
--tserver_master_addrs=127.1.166.62:45921
--webserver_port=40015
--webserver_interface=127.1.166.3
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:00:09 dist-test-slave-rdl9 krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:09.717517 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.707142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:50919 (local address 127.1.166.62:45921)
0620 23:00:09.707193 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.707195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.707423 (+ 228us) server_negotiation.cc:407] Connection header received
0620 23:00:09.707629 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.707630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.707645 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.707696 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:09.708327 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.708889 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.710076 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.710389 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.710458 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.713826 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:09.713828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:09.713829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:09.713840 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:09.715687 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.716224 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.716225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.716226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.716307 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.716711 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.716712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.716712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.716891 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:09.716972 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.717299 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.717410 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:00:09.717741 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.707081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:50919)
0620 23:00:09.707272 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.707334 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:09.707546 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:09.707750 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:09.707754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:09.708028 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:09.708240 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.708245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.709070 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.709071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.709890 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.709893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.710471 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.710472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.710616 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.711725 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:09.711736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:09.713686 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:09.715825 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:09.715833 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:09.715836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:09.716127 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:09.716455 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:09.716457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:09.716458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:09.716631 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:09.716988 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:09.716995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:09.717149 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 23:00:09.717256 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:00:09.717431 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260620 23:00:09.718330 17961 heartbeater.cc:347] Connected to a master server at 127.1.166.62:45921
I20260620 23:00:09.718478 17961 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:09.718698 17961 heartbeater.cc:511] Master 127.1.166.62:45921 requested a full tablet report, sending...
I20260620 23:00:09.719378 17632 ts_manager.cc:194] Registered new tserver with Master: a574ffe02c2c4e1fa1a099e965bbd0ea (127.1.166.2:40103)
I20260620 23:00:09.719969 17632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:50919
Jun 20 23:00:09 dist-test-slave-rdl9 krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.2@KRBTEST.COM
I20260620 23:00:09.729987 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.719699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:40103 (local address 127.0.0.1:36058)
0620 23:00:09.719750 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.719760 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:09.719794 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:09.720128 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:09.720129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:09.720518 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:09.720565 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.720567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.721965 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.721967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.722584 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.722587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.723301 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.723302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.723354 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.723994 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:09.724000 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:09.725742 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:09.728275 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:09.728294 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:09.728298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:09.728644 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:09.728973 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:09.728973 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:09.728974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:09.729053 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:09.729555 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:09.729566 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:09.729647 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 23:00:09.729733 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:00:09.729872 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 23:00:09.730232 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.719781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.1.166.2:40103)
0620 23:00:09.719977 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.719983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.719992 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:00:09.720030 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.720032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.720048 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.720122 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:09.720657 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.721802 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.722665 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.723200 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.723290 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.725970 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:09.725988 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:09.725988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:09.726000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:09.728126 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.728782 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.728786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.728787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.728881 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.729222 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.729223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.729224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.729448 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:09.729567 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.729841 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.729985 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260620 23:00:09.730823 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet f10d1f784b1941e1bad7639c9cfe3f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:59:49 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:36058
I20260620 23:00:09.731628 17966 tablet_replica.cc:330] stopping tablet replica
I20260620 23:00:09.731765 17966 raft_consensus.cc:2229] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:09.731854 17966 raft_consensus.cc:2258] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:09.732267 17966 ts_tablet_manager.cc:1905] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:09.734609 17966 ts_tablet_manager.cc:1918] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:09.734678 17966 log.cc:1196] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/f10d1f784b1941e1bad7639c9cfe3f5e
I20260620 23:00:09.734886 17966 ts_tablet_manager.cc:1939] T f10d1f784b1941e1bad7639c9cfe3f5e P a574ffe02c2c4e1fa1a099e965bbd0ea: Deleting consensus metadata
I20260620 23:00:09.735562 17616 catalog_manager.cc:4841] TS a574ffe02c2c4e1fa1a099e965bbd0ea (127.1.166.2:40103): tablet f10d1f784b1941e1bad7639c9cfe3f5e (table test-table [id=10c1992fc57246f28c5dffdd5748b832]) successfully deleted
W20260620 23:00:09.803627 17964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:09.803853 17964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:09.803887 17964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:09.805299 17964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:09.805361 17964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:09.805404 17964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 23:00:09.807395 17964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:35583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=40015
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:09.807914 17964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:09.808208 17964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:09.814934 17974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:09.814955 17964 server_base.cc:1034] running on GCE node
W20260620 23:00:09.814934 17971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:09.814962 17972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:09.815337 17964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:09.815615 17964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:09.816790 17964 hybrid_clock.cc:648] HybridClock initialized: now 1781996409816783 us; error 33 us; skew 500 ppm
Jun 20 23:00:09 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:09.819685 17964 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 23:00:09.820143 17964 webserver.cc:460] Webserver started at http://127.1.166.3:40015/ using document root <none> and password file <none>
I20260620 23:00:09.820369 17964 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:09.820441 17964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:09.821890 17964 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:09.822662 17981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:09.822996 17964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:09.823168 17964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "38c8f5e7b4a64d419d0bf956b139dc83"
format_stamp: "Formatted at 2026-06-20 22:59:49 on dist-test-slave-rdl9"
server_key: "10cdf0204ac3a21b5f5df233792d3040"
server_key_iv: "665b86dab2e925d814274d1028445d43"
server_key_version: "encryptionkey@0"
I20260620 23:00:09.823727 17964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:09.835731 17964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:09.837951 17964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:09.838073 17964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:09.838264 17964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:09.838624 17964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:09.838658 17964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:09.838681 17964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:09.838693 17964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:09.844521 17964 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:35583
I20260620 23:00:09.844565 18094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:35583 every 8 connection(s)
I20260620 23:00:09.844921 17964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:00:09 dist-test-slave-rdl9 krb5kdc[17052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:09.854601 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 17964
I20260620 23:00:09.856437 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.845766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:44371 (local address 127.1.166.62:45921)
0620 23:00:09.845884 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.845887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.846086 (+ 199us) server_negotiation.cc:407] Connection header received
0620 23:00:09.846819 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.846821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.846836 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.846892 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:09.848080 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.848646 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.849819 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.850033 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.850094 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.852886 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:09.852888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:09.852888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:09.852898 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:09.854862 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.855479 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.855480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.855480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.855540 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.855779 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.855780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.855780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.855904 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:09.855959 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.856255 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.856343 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:00:09.856647 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.845639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:44371)
0620 23:00:09.845895 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.845969 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:09.846690 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:09.846983 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:09.846987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:09.847719 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:09.847971 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.847978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.848736 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.848737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.849735 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.849738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.850107 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.850108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.850245 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.850996 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:09.851008 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:09.852776 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:09.854983 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:09.855000 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:09.855003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:09.855390 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:09.855596 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:09.855596 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:09.855597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:09.855715 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:09.855958 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:09.855964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:09.856125 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 23:00:09.856219 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:00:09.856411 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260620 23:00:09.857250 18095 heartbeater.cc:347] Connected to a master server at 127.1.166.62:45921
I20260620 23:00:09.857359 18095 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:09.857550 18095 heartbeater.cc:511] Master 127.1.166.62:45921 requested a full tablet report, sending...
I20260620 23:00:09.858024 17632 ts_manager.cc:194] Registered new tserver with Master: 38c8f5e7b4a64d419d0bf956b139dc83 (127.1.166.3:35583)
I20260620 23:00:09.858489 17632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:44371
I20260620 23:00:09.862002 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.855092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.1.166.62:45921)
0620 23:00:09.855225 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.855227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.855345 (+ 118us) server_negotiation.cc:407] Connection header received
0620 23:00:09.855382 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.855383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.855395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.855486 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:09.856068 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.856637 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.857618 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.857899 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.857961 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.859357 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:09.859359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:09.859359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:09.859365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:09.860758 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.861188 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.861189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.861190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.861251 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:09.861477 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:09.861478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:09.861478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:09.861620 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:09.861754 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.861841 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.861920 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260620 23:00:09.862747 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:09.865541 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.863141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.1.166.62:45921)
0620 23:00:09.863192 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.863193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.863205 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:00:09.863277 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.863277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.863307 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.863363 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:00:09.863457 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.863924 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.864727 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.864929 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.864992 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.865169 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:00:09.865287 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:00:09.865363 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.865445 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.865471 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:00:09.865832 17632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:00:09.867503 17632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:09.870504 17894 tablet_service.cc:1467] Processing CreateTablet for tablet 813de076b17c4bbc86d212119ec75328 (DEFAULT_TABLE table=test-table [id=5d3cdd82d39d4268879784896713e0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:09.870615 17754 tablet_service.cc:1467] Processing CreateTablet for tablet 813de076b17c4bbc86d212119ec75328 (DEFAULT_TABLE table=test-table [id=5d3cdd82d39d4268879784896713e0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:09.870886 17754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813de076b17c4bbc86d212119ec75328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:09.870886 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813de076b17c4bbc86d212119ec75328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:09.872855 17712 tablet_bootstrap.cc:492] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: Bootstrap starting.
I20260620 23:00:09.872843 17851 tablet_bootstrap.cc:492] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: Bootstrap starting.
I20260620 23:00:09.873315 17712 tablet_bootstrap.cc:654] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:09.873323 17851 tablet_bootstrap.cc:654] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:09.873972 17712 tablet_bootstrap.cc:492] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: No bootstrap required, opened a new log
I20260620 23:00:09.874023 17712 ts_tablet_manager.cc:1397] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:09.874084 17851 tablet_bootstrap.cc:492] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: No bootstrap required, opened a new log
I20260620 23:00:09.874145 17851 ts_tablet_manager.cc:1397] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:09.874145 17712 raft_consensus.cc:348] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.874236 17712 raft_consensus.cc:374] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:09.874311 17712 raft_consensus.cc:729] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f1e86860dc4fa2a626b0f4b3f2ba80, State: Initialized, Role: FOLLOWER
I20260620 23:00:09.874396 17712 consensus_queue.cc:260] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.874500 17712 ts_tablet_manager.cc:1428] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 23:00:09.874490 17851 raft_consensus.cc:348] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.874588 17851 raft_consensus.cc:374] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:09.874660 17851 raft_consensus.cc:729] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a574ffe02c2c4e1fa1a099e965bbd0ea, State: Initialized, Role: FOLLOWER
I20260620 23:00:09.874716 17851 consensus_queue.cc:260] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.874804 17851 ts_tablet_manager.cc:1428] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:09.874898 17961 heartbeater.cc:503] Master 127.1.166.62:45921 was elected leader, sending a full tablet report...
I20260620 23:00:09.876504 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.870020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:35583 (local address 127.0.0.1:38564)
0620 23:00:09.870108 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.870122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:09.870184 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:09.873186 (+ 3002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:09.873188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:09.873192 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:09.873280 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.873282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.874533 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.874535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.875533 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.875536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.876250 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.876251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.876315 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.876319 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:00:09.876374 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 23:00:09.876427 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 23:00:09.876518 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.870190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.1.166.3:35583)
0620 23:00:09.872884 (+ 2694us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.872891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.872906 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:00:09.872957 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.872960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.872980 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.873201 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:09.873359 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.874421 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.875626 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.876164 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.876232 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.876303 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.876390 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.876433 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2640,"thread_start_us":184,"threads_started":1}
I20260620 23:00:09.877278 18029 tablet_service.cc:1467] Processing CreateTablet for tablet 813de076b17c4bbc86d212119ec75328 (DEFAULT_TABLE table=test-table [id=5d3cdd82d39d4268879784896713e0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:09.877694 18029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813de076b17c4bbc86d212119ec75328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:09.877821 17826 raft_consensus.cc:482] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:09.877924 17826 raft_consensus.cc:504] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.878278 17826 leader_election.cc:288] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38c8f5e7b4a64d419d0bf956b139dc83 (127.1.166.3:35583), a574ffe02c2c4e1fa1a099e965bbd0ea (127.1.166.2:40103)
I20260620 23:00:09.881167 18105 tablet_bootstrap.cc:492] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: Bootstrap starting.
I20260620 23:00:09.881906 18105 tablet_bootstrap.cc:654] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:09.881996 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.878428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:35583 (local address 127.1.166.1:47871)
0620 23:00:09.878653 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.878663 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:09.878717 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:09.878840 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:09.878840 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:09.878842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:09.879156 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.879158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.880001 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.880002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.880980 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.880983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.881710 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.881712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.881781 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.881786 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:09.881851 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:00:09.881898 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260620 23:00:09.882009 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.878484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:47871 (local address 127.1.166.3:35583)
0620 23:00:09.878529 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.878531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.878709 (+ 178us) server_negotiation.cc:407] Connection header received
0620 23:00:09.878757 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.878758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.878767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.878837 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:09.879227 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.879905 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.881063 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.881610 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.881675 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.881686 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.881866 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.881903 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:00:09.882687 18105 log.cc:824] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:09.882414 18049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813de076b17c4bbc86d212119ec75328" candidate_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" is_pre_election: true
I20260620 23:00:09.883381 18105 tablet_bootstrap.cc:492] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: No bootstrap required, opened a new log
I20260620 23:00:09.883450 18105 ts_tablet_manager.cc:1397] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
W20260620 23:00:09.883628 17706 leader_election.cc:341] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 38c8f5e7b4a64d419d0bf956b139dc83 (127.1.166.3:35583): Illegal state: must be running to vote when last-logged opid is not known
I20260620 23:00:09.884531 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.878515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:40103 (local address 127.1.166.1:33413)
0620 23:00:09.881206 (+ 2691us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.881226 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:09.881280 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:09.881469 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:09.881470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:09.881472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:09.881535 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.881536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.882326 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.882327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.883386 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:09.883389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.884288 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.884289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:09.884353 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.884356 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:09.884402 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 23:00:09.884459 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2671,"thread_start_us":92,"threads_started":1}
I20260620 23:00:09.884542 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.878606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:33413 (local address 127.1.166.2:40103)
0620 23:00:09.878646 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.878648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.881330 (+ 2682us) server_negotiation.cc:407] Connection header received
0620 23:00:09.881359 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.881360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.881381 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.881430 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:09.881597 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.882237 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.883484 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.884122 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.884179 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.884243 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.884425 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.884467 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:00:09.884569 18105 raft_consensus.cc:348] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.884734 18105 raft_consensus.cc:374] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:09.884763 18105 raft_consensus.cc:729] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c8f5e7b4a64d419d0bf956b139dc83, State: Initialized, Role: FOLLOWER
I20260620 23:00:09.884864 17914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813de076b17c4bbc86d212119ec75328" candidate_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" is_pre_election: true
I20260620 23:00:09.884893 18105 consensus_queue.cc:260] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.884991 17914 raft_consensus.cc:2454] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41f1e86860dc4fa2a626b0f4b3f2ba80 in term 0.
I20260620 23:00:09.885182 18105 ts_tablet_manager.cc:1428] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:00:09.885175 17706 leader_election.cc:302] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41f1e86860dc4fa2a626b0f4b3f2ba80, a574ffe02c2c4e1fa1a099e965bbd0ea; no voters: 38c8f5e7b4a64d419d0bf956b139dc83
I20260620 23:00:09.885233 18095 heartbeater.cc:503] Master 127.1.166.62:45921 was elected leader, sending a full tablet report...
I20260620 23:00:09.885303 17826 raft_consensus.cc:2781] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:09.885349 17826 raft_consensus.cc:482] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:09.885370 17826 raft_consensus.cc:3037] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:09.886260 17826 raft_consensus.cc:504] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.886417 17826 leader_election.cc:288] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [CANDIDATE]: Term 1 election: Requested vote from peers 38c8f5e7b4a64d419d0bf956b139dc83 (127.1.166.3:35583), a574ffe02c2c4e1fa1a099e965bbd0ea (127.1.166.2:40103)
I20260620 23:00:09.886596 18049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813de076b17c4bbc86d212119ec75328" candidate_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c8f5e7b4a64d419d0bf956b139dc83"
I20260620 23:00:09.886597 17914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813de076b17c4bbc86d212119ec75328" candidate_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea"
I20260620 23:00:09.886669 18049 raft_consensus.cc:3037] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:09.886669 17914 raft_consensus.cc:3037] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:09.887295 18049 raft_consensus.cc:2454] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f1e86860dc4fa2a626b0f4b3f2ba80 in term 1.
I20260620 23:00:09.887418 17914 raft_consensus.cc:2454] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f1e86860dc4fa2a626b0f4b3f2ba80 in term 1.
I20260620 23:00:09.887468 17706 leader_election.cc:302] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c8f5e7b4a64d419d0bf956b139dc83, 41f1e86860dc4fa2a626b0f4b3f2ba80; no voters:
I20260620 23:00:09.887540 17826 raft_consensus.cc:2781] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:09.887669 17826 raft_consensus.cc:686] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 LEADER]: Becoming Leader. State: Replica: 41f1e86860dc4fa2a626b0f4b3f2ba80, State: Running, Role: LEADER
I20260620 23:00:09.887735 17826 consensus_queue.cc:237] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } }
I20260620 23:00:09.888422 17630 catalog_manager.cc:5495] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41f1e86860dc4fa2a626b0f4b3f2ba80 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 36801 } health_report { overall_health: HEALTHY } } }
I20260620 23:00:09.898876 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.896561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.1.166.1:36801)
0620 23:00:09.896621 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:09.896623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:09.896630 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:00:09.896670 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:09.896671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:09.896698 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:09.896766 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:00:09.896883 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.897365 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.898112 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:09.898304 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:09.898382 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:09.898542 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:00:09.898651 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:00:09.898715 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:09.898753 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 23:00:09.898783 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:00:09.900445 18049 raft_consensus.cc:1261] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 FOLLOWER]: Refusing update from remote peer 41f1e86860dc4fa2a626b0f4b3f2ba80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:09.900450 17914 raft_consensus.cc:1261] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Refusing update from remote peer 41f1e86860dc4fa2a626b0f4b3f2ba80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:09.900771 17826 consensus_queue.cc:1035] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c8f5e7b4a64d419d0bf956b139dc83" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:09.900921 18108 consensus_queue.cc:1035] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:09.902940 18111 mvcc.cc:205] Tried to move back new op lower bound from 7299057294949732352 to 7299057294900695040. Current Snapshot: MvccSnapshot[applied={T|T < 7299057294949732352}]
I20260620 23:00:09.903218 18113 mvcc.cc:205] Tried to move back new op lower bound from 7299057294949732352 to 7299057294900695040. Current Snapshot: MvccSnapshot[applied={T|T < 7299057294949732352}]
I20260620 23:00:09.905689 17632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:09.905830 17632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:09.906935 17632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Sending DeleteTablet for 3 replicas of tablet 813de076b17c4bbc86d212119ec75328
I20260620 23:00:09.907351 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet 813de076b17c4bbc86d212119ec75328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:09 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:36058
I20260620 23:00:09.907410 17754 tablet_service.cc:1514] Processing DeleteTablet for tablet 813de076b17c4bbc86d212119ec75328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:09 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:45614
I20260620 23:00:09.907382 18029 tablet_service.cc:1514] Processing DeleteTablet for tablet 813de076b17c4bbc86d212119ec75328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:09 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:38564
I20260620 23:00:09.907461 17966 tablet_replica.cc:330] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: stopping tablet replica
I20260620 23:00:09.907495 17829 tablet_replica.cc:330] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: stopping tablet replica
I20260620 23:00:09.907657 17829 raft_consensus.cc:2229] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:00:09.907639 17966 raft_consensus.cc:2229] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:09.907745 17966 raft_consensus.cc:2258] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:09.907783 17829 raft_consensus.cc:2258] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:09.907990 18121 tablet_replica.cc:330] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: stopping tablet replica
I20260620 23:00:09.908186 18121 raft_consensus.cc:2229] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:09.908221 17966 ts_tablet_manager.cc:1905] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:09.908268 17829 ts_tablet_manager.cc:1905] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:09.908430 18121 raft_consensus.cc:2258] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:09.909126 18121 ts_tablet_manager.cc:1905] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:09.909714 17829 ts_tablet_manager.cc:1918] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:09.909714 17966 ts_tablet_manager.cc:1918] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:09.909778 17966 log.cc:1196] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/813de076b17c4bbc86d212119ec75328
I20260620 23:00:09.909778 17829 log.cc:1196] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal/wals/813de076b17c4bbc86d212119ec75328
I20260620 23:00:09.909972 17966 ts_tablet_manager.cc:1939] T 813de076b17c4bbc86d212119ec75328 P a574ffe02c2c4e1fa1a099e965bbd0ea: Deleting consensus metadata
I20260620 23:00:09.909972 17829 ts_tablet_manager.cc:1939] T 813de076b17c4bbc86d212119ec75328 P 41f1e86860dc4fa2a626b0f4b3f2ba80: Deleting consensus metadata
I20260620 23:00:09.910329 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 17670
I20260620 23:00:09.910472 17616 catalog_manager.cc:4841] TS a574ffe02c2c4e1fa1a099e965bbd0ea (127.1.166.2:40103): tablet 813de076b17c4bbc86d212119ec75328 (table test-table [id=5d3cdd82d39d4268879784896713e0bd]) successfully deleted
I20260620 23:00:09.911470 18121 ts_tablet_manager.cc:1918] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:09.911613 18121 log.cc:1196] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/813de076b17c4bbc86d212119ec75328
I20260620 23:00:09.911902 18121 ts_tablet_manager.cc:1939] T 813de076b17c4bbc86d212119ec75328 P 38c8f5e7b4a64d419d0bf956b139dc83: Deleting consensus metadata
I20260620 23:00:09.912360 17616 catalog_manager.cc:4841] TS 38c8f5e7b4a64d419d0bf956b139dc83 (127.1.166.3:35583): tablet 813de076b17c4bbc86d212119ec75328 (table test-table [id=5d3cdd82d39d4268879784896713e0bd]) successfully deleted
W20260620 23:00:09.916944 17616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:36801 (error 108)
I20260620 23:00:09.917394 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 17825
I20260620 23:00:09.917591 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:09.917452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:36801 (local address 127.0.0.1:45636)
0620 23:00:09.917510 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:09.917546 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:36801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:00:09.919276 17616 catalog_manager.cc:4584] TS 41f1e86860dc4fa2a626b0f4b3f2ba80 (127.1.166.1:36801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 813de076b17c4bbc86d212119ec75328: Network error: Client connection negotiation failed: client connection to 127.1.166.1:36801: connect: Connection refused (error 111)
I20260620 23:00:09.922919 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 17964
I20260620 23:00:09.929214 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 17599
I20260620 23:00:29.935075 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45921
--webserver_interface=127.1.166.62
--webserver_port=37141
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:45921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:30.031581 18123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:30.031893 18123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:30.031922 18123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:30.033416 18123 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:00:30.033469 18123 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:00:30.033483 18123 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:00:30.033492 18123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:30.033505 18123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:30.033519 18123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:30.033530 18123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:30.033541 18123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:30.035706 18123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:45921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:30.036075 18123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:30.036360 18123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:30.042204 18131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:30.042246 18129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:30.042204 18128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:30.042294 18123 server_base.cc:1034] running on GCE node
I20260620 23:00:30.042690 18123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:30.043000 18123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:30.044164 18123 hybrid_clock.cc:648] HybridClock initialized: now 1781996430044148 us; error 31 us; skew 500 ppm
Jun 20 23:00:30 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:30.047605 18123 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:00:30.048264 18123 webserver.cc:460] Webserver started at http://127.1.166.62:37141/ using document root <none> and password file <none>
I20260620 23:00:30.048521 18123 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:30.048574 18123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:30.050220 18123 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:00:30.051149 18138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.051469 18123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:30.051658 18123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "dc6990539d3a4f70979d534e8a5a933d"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "894ff1a4506225367fc5daba0688475f"
server_key_iv: "bf3de0da63f7967abab9505a664377f4"
server_key_version: "encryptionkey@0"
I20260620 23:00:30.052237 18123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:30.071067 18123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:30.074553 18123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:30.074766 18123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:30.081971 18123 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:45921
I20260620 23:00:30.081993 18190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:45921 every 8 connection(s)
I20260620 23:00:30.082587 18123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:30.084282 18191 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:00:30.084982 18191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Bootstrap starting.
I20260620 23:00:30.092536 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 18123
I20260620 23:00:30.092900 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:36801
--local_ip_for_outbound_sockets=127.1.166.1
--tserver_master_addrs=127.1.166.62:45921
--webserver_port=46859
--webserver_interface=127.1.166.1
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:00:30.093477 18191 log.cc:824] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:30.096275 18191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:00:30.096560 18191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Bootstrap complete.
I20260620 23:00:30.098722 18191 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 23:00:30.098955 18191 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6990539d3a4f70979d534e8a5a933d, State: Initialized, Role: FOLLOWER
I20260620 23:00:30.099105 18191 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 23:00:30.099179 18191 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:30.099226 18191 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:30.099273 18191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:00:30.100267 18191 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 23:00:30.100368 18191 leader_election.cc:302] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6990539d3a4f70979d534e8a5a933d; no voters:
I20260620 23:00:30.100565 18191 leader_election.cc:288] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:00:30.100611 18195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:00:30.100739 18195 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [term 3 LEADER]: Becoming Leader. State: Replica: dc6990539d3a4f70979d534e8a5a933d, State: Running, Role: LEADER
I20260620 23:00:30.100824 18195 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } }
I20260620 23:00:30.100869 18191 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:30.101176 18196 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dc6990539d3a4f70979d534e8a5a933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } } }
I20260620 23:00:30.101265 18196 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:30.101514 18197 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6990539d3a4f70979d534e8a5a933d. Latest consensus state: current_term: 3 leader_uuid: "dc6990539d3a4f70979d534e8a5a933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6990539d3a4f70979d534e8a5a933d" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45921 } } }
I20260620 23:00:30.101639 18197 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:30.101769 18201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:30.102777 18201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:30.102900 18201 catalog_manager.cc:1259] Loaded cluster ID: 4630d6da8f2e4420bf08823e5520baf7
I20260620 23:00:30.102969 18201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:30.104355 18201 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:30.104589 18201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Loaded TSK: 0
I20260620 23:00:30.104641 18201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc6990539d3a4f70979d534e8a5a933d: Loaded TSK: 1
F20260620 23:00:30.104897 18201 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd7928819ad google::LogMessage::Fail() at ??:0
@ 0x7fd79288488a google::LogMessage::SendToLog() at ??:0
@ 0x7fd7928814de google::LogMessage::Flush() at ??:0
@ 0x7fd792882a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd794fee104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd794febd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd79a7db1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd79a7db48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd79a7db594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd79a7a1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd79a7abad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd79474f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd794744954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd7955a86db start_thread at ??:0
@ 0x7fd79137f61f clone at ??:0
W20260620 23:00:30.203069 18194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:30.203307 18194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:30.203334 18194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:30.204663 18194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:30.204722 18194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:30.204753 18194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 23:00:30.206811 18194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:36801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=46859
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:30.207228 18194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:30.207551 18194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:30.215092 18233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:30.215152 18232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:30.215222 18235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:30.216099 18194 server_base.cc:1034] running on GCE node
I20260620 23:00:30.216363 18194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:30.216643 18194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:30.217921 18194 hybrid_clock.cc:648] HybridClock initialized: now 1781996430217885 us; error 52 us; skew 500 ppm
Jun 20 23:00:30 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:30.221073 18194 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 23:00:30.221604 18194 webserver.cc:460] Webserver started at http://127.1.166.1:46859/ using document root <none> and password file <none>
I20260620 23:00:30.221820 18194 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:30.221870 18194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:30.223415 18194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:30.224025 18242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.224198 18194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:00:30.224282 18194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "41f1e86860dc4fa2a626b0f4b3f2ba80"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "484d79d194e20508c2450699f41f0c74"
server_key_iv: "550d7ca30d3a98a3a6ade859e8b1e954"
server_key_version: "encryptionkey@0"
I20260620 23:00:30.224643 18194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:30.234691 18194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:30.237440 18194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:30.237600 18194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:30.237821 18194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:30.238127 18194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:30.238166 18194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.238197 18194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:30.238221 18194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.243886 18194 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:36801
I20260620 23:00:30.244021 18355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:36801 every 8 connection(s)
I20260620 23:00:30.244316 18194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 23:00:30.249105 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 18194
I20260620 23:00:30.249436 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:40103
--local_ip_for_outbound_sockets=127.1.166.2
--tserver_master_addrs=127.1.166.62:45921
--webserver_port=36811
--webserver_interface=127.1.166.2
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:30.374094 18358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:00:30.245047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:57247)
0620 23:00:30.245216 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.245259 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:30.245722 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:30.373563 (+127841us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
W20260620 23:00:30.374657 18356 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:45921 (0 consecutive failures): Network error: Failed to ping master at 127.1.166.62:45921: Client connection negotiation failed: client connection to 127.1.166.62:45921: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:00:30.375270 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:30.374998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:45769)
0620 23:00:30.375098 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.375175 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:00:30.375873 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:30.375762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:47945)
0620 23:00:30.375804 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.375838 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 23:00:30.376009 18356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:00:30.378970 18359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:30.379225 18359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:30.379261 18359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:30.380645 18359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:30.380702 18359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:30.380743 18359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 23:00:30.382544 18359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:40103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=36811
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:30.382823 18359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:30.383071 18359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:30.388092 18364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:30.388094 18367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:30.388094 18365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:30.388391 18359 server_base.cc:1034] running on GCE node
I20260620 23:00:30.388602 18359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:30.389647 18359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:30.390875 18359 hybrid_clock.cc:648] HybridClock initialized: now 1781996430390791 us; error 96 us; skew 500 ppm
Jun 20 23:00:30 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:30.393613 18359 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 23:00:30.394120 18359 webserver.cc:460] Webserver started at http://127.1.166.2:36811/ using document root <none> and password file <none>
I20260620 23:00:30.394330 18359 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:30.394423 18359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:30.395841 18359 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:30.396500 18374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.396672 18359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:30.396770 18359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "a574ffe02c2c4e1fa1a099e965bbd0ea"
format_stamp: "Formatted at 2026-06-20 22:59:48 on dist-test-slave-rdl9"
server_key: "3c5d366ef0884ddc65df5c9b807c2acd"
server_key_iv: "5b6286dc9b0c94ab9584f4f2e5c4c7bd"
server_key_version: "encryptionkey@0"
I20260620 23:00:30.397131 18359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:30.414248 18359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:30.416715 18359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:30.416888 18359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:30.417145 18359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:30.417472 18359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:30.417510 18359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.417546 18359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:30.417574 18359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.424356 18359 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:40103
I20260620 23:00:30.424404 18487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:40103 every 8 connection(s)
I20260620 23:00:30.424858 18359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 23:00:30.426000 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:30.425573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:50697)
0620 23:00:30.425768 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.425844 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
W20260620 23:00:30.426539 18488 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:45921 (0 consecutive failures): Network error: Failed to ping master at 127.1.166.62:45921: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
I20260620 23:00:30.426693 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 18359
I20260620 23:00:30.426874 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:30.426754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:43591)
0620 23:00:30.426808 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.426846 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 23:00:30.426975 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:35583
--local_ip_for_outbound_sockets=127.1.166.3
--tserver_master_addrs=127.1.166.62:45921
--webserver_port=40015
--webserver_interface=127.1.166.3
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:00:30.427234 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:30.427135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:43837)
0620 23:00:30.427180 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.427207 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 23:00:30.427359 18488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:00:30.517719 18491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:30.518021 18491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:30.518056 18491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:30.519760 18491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:30.519840 18491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:30.519873 18491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 23:00:30.521971 18491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:35583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=40015
--tserver_master_addrs=127.1.166.62:45921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:30.522291 18491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:30.522576 18491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:30.527716 18497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:30.527732 18496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:30.527732 18499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:30.528105 18491 server_base.cc:1034] running on GCE node
I20260620 23:00:30.528306 18491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:30.528599 18491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:30.529852 18491 hybrid_clock.cc:648] HybridClock initialized: now 1781996430529823 us; error 35 us; skew 500 ppm
Jun 20 23:00:30 dist-test-slave-rdl9 krb5kdc[17052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:30.533553 18491 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 23:00:30.534155 18491 webserver.cc:460] Webserver started at http://127.1.166.3:40015/ using document root <none> and password file <none>
I20260620 23:00:30.534483 18491 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:30.534547 18491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:30.536450 18491 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:00:30.537417 18506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.537712 18491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:30.537827 18491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "38c8f5e7b4a64d419d0bf956b139dc83"
format_stamp: "Formatted at 2026-06-20 22:59:49 on dist-test-slave-rdl9"
server_key: "10cdf0204ac3a21b5f5df233792d3040"
server_key_iv: "665b86dab2e925d814274d1028445d43"
server_key_version: "encryptionkey@0"
I20260620 23:00:30.538275 18491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:30.548094 18491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:30.551056 18491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:30.551218 18491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:30.551467 18491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:30.551800 18491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:30.551834 18491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.551867 18491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:30.551891 18491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:30.558398 18491 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:35583
I20260620 23:00:30.558470 18619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:35583 every 8 connection(s)
I20260620 23:00:30.558889 18491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 23:00:30.560540 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:30.559768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:57251)
0620 23:00:30.560034 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.560140 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":110,"threads_started":1}
W20260620 23:00:30.560851 18620 heartbeater.cc:644] Failed to heartbeat to 127.1.166.62:45921 (0 consecutive failures): Network error: Failed to ping master at 127.1.166.62:45921: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
I20260620 23:00:30.561272 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:30.561182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:41887)
0620 23:00:30.561222 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.561252 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 23:00:30.561529 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:30.561467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:40557)
0620 23:00:30.561494 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:30.561515 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 23:00:30.561640 18620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:00:30.563660 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 18491
I20260620 23:00:31.376786 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:31.376507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:58267)
0620 23:00:31.376667 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:31.376716 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260620 23:00:31.428054 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:31.427768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:38645)
0620 23:00:31.427927 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:31.427975 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260620 23:00:31.562368 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:31.562091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:39091)
0620 23:00:31.562234 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:31.562278 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260620 23:00:32.377594 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:32.377326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:33355)
0620 23:00:32.377489 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:32.377538 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260620 23:00:32.428853 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:32.428608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:50321)
0620 23:00:32.428760 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:32.428804 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260620 23:00:32.563252 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:32.562955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:58485)
0620 23:00:32.563148 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:32.563198 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260620 23:00:33.378455 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:33.378142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:60325)
0620 23:00:33.378329 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:33.378401 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260620 23:00:33.429869 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:33.429499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:39539)
0620 23:00:33.429727 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:33.429799 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260620 23:00:33.564128 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:33.563847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:52565)
0620 23:00:33.564028 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:33.564073 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260620 23:00:34.379418 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:34.379137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:60515)
0620 23:00:34.379318 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:34.379368 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260620 23:00:34.431241 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:34.430682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:36145)
0620 23:00:34.430939 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:34.431168 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260620 23:00:34.564968 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:34.564693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:58731)
0620 23:00:34.564870 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:34.564911 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260620 23:00:35.380326 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:35.380024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:45473)
0620 23:00:35.380208 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:35.380273 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260620 23:00:35.432337 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:35.431926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:58195)
0620 23:00:35.432186 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:35.432269 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260620 23:00:35.565974 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:35.565624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:46753)
0620 23:00:35.565810 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:35.565860 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260620 23:00:36.381258 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:36.380970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:38045)
0620 23:00:36.381149 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:36.381200 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260620 23:00:36.433312 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:36.432988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:33339)
0620 23:00:36.433195 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:36.433261 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260620 23:00:36.567098 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:36.566789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:44537)
0620 23:00:36.566972 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:36.567033 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
I20260620 23:00:37.382045 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:37.381809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:47545)
0620 23:00:37.381955 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:37.381998 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260620 23:00:37.434145 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:37.433836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:56119)
0620 23:00:37.434032 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:37.434070 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260620 23:00:37.568137 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:37.567760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:55799)
0620 23:00:37.568000 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:37.568069 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
I20260620 23:00:38.382923 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:38.382661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:46077)
0620 23:00:38.382822 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:38.382873 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260620 23:00:38.435321 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:38.434941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:54223)
0620 23:00:38.435190 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:38.435265 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":135,"threads_started":1}
I20260620 23:00:38.569238 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:38.568789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:37817)
0620 23:00:38.569079 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:38.569164 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260620 23:00:39.384006 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:39.383668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:53827)
0620 23:00:39.383898 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:39.383955 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
I20260620 23:00:39.436246 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:39.435997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:44109)
0620 23:00:39.436160 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:39.436197 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260620 23:00:39.570295 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:39.569973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:54059)
0620 23:00:39.570180 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:39.570235 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260620 23:00:40.384896 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:40.384635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:39173)
0620 23:00:40.384800 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:40.384848 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260620 23:00:40.437247 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:40.436930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:51945)
0620 23:00:40.437132 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:40.437179 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260620 23:00:40.571555 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:40.571056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:52473)
0620 23:00:40.571229 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:40.571381 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260620 23:00:41.385993 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:41.385620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:32995)
0620 23:00:41.385857 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:41.385916 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
I20260620 23:00:41.438170 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:41.437817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:52443)
0620 23:00:41.438022 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:41.438098 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260620 23:00:41.572650 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:41.572288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:42401)
0620 23:00:41.572467 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:41.572543 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260620 23:00:42.387502 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:42.387112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:58527)
0620 23:00:42.387346 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:42.387415 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
I20260620 23:00:42.439208 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:42.438869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:33831)
0620 23:00:42.439073 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:42.439152 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
I20260620 23:00:42.573836 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:42.573482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:42491)
0620 23:00:42.573707 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:42.573765 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
I20260620 23:00:43.388476 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:43.388187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:54291)
0620 23:00:43.388365 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:43.388418 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260620 23:00:43.440068 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:43.439811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:45857)
0620 23:00:43.439962 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:43.440011 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260620 23:00:43.574805 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:43.574522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:45285)
0620 23:00:43.574700 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:43.574752 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260620 23:00:44.389586 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:44.389296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:60761)
0620 23:00:44.389480 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:44.389531 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260620 23:00:44.440959 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:44.440678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:44859)
0620 23:00:44.440858 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:44.440900 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260620 23:00:44.575997 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:44.575657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:45061)
0620 23:00:44.575876 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:44.575943 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":111,"threads_started":1}
I20260620 23:00:45.390646 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:45.390424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.1:48017)
0620 23:00:45.390548 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:45.390593 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260620 23:00:45.441995 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:45.441686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.2:33373)
0620 23:00:45.441880 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:45.441944 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":106,"threads_started":1}
I20260620 23:00:45.573344 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 18194
I20260620 23:00:45.577178 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:45.576813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:45921 (local address 127.1.166.3:57963)
0620 23:00:45.577064 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:45.577119 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.62:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260620 23:00:45.578637 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 18359
I20260620 23:00:45.583340 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 18491
2026-06-20T23:00:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58262 ms)
[----------] 30 tests from SecurityITest (269826 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:00:45 dist-test-slave-rdl9 krb5kdc[18671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:00:45 dist-test-slave-rdl9 krb5kdc[18671](info): set up 2 sockets
Jun 20 23:00:45 dist-test-slave-rdl9 krb5kdc[18671](info): commencing operation
krb5kdc: starting...
W20260620 23:00:46.589841 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:00:46 dist-test-slave-rdl9 krb5kdc[18671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:00:46.721230 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:44205
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:44205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:46.807920 18684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:46.808190 18684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:46.808228 18684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:46.809585 18684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:46.809651 18684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:46.809671 18684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:46.809689 18684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:46.809705 18684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:46.811503 18684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:44205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:46.811808 18684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:46.812058 18684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:46.818236 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:46.818389 18684 server_base.cc:1034] running on GCE node
W20260620 23:00:46.818239 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:46.818291 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:46.818955 18684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:46.819321 18684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:46.820566 18684 hybrid_clock.cc:648] HybridClock initialized: now 1781996446820542 us; error 45 us; skew 500 ppm
Jun 20 23:00:46 dist-test-slave-rdl9 krb5kdc[18671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996446, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:46.824049 18684 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:00:46.824643 18684 webserver.cc:460] Webserver started at http://127.1.166.62:39477/ using document root <none> and password file <none>
I20260620 23:00:46.824883 18684 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:46.824936 18684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:46.825053 18684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:46.826304 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "4d2a5209db464e49af94408b3c735186"
format_stamp: "Formatted at 2026-06-20 23:00:46 on dist-test-slave-rdl9"
server_key: "74e5289ef7a6881d5a5cc652d7f097c5"
server_key_iv: "23c1820fd6a39560ad4c301301978689"
server_key_version: "encryptionkey@0"
I20260620 23:00:46.826745 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "4d2a5209db464e49af94408b3c735186"
format_stamp: "Formatted at 2026-06-20 23:00:46 on dist-test-slave-rdl9"
server_key: "74e5289ef7a6881d5a5cc652d7f097c5"
server_key_iv: "23c1820fd6a39560ad4c301301978689"
server_key_version: "encryptionkey@0"
I20260620 23:00:46.828323 18684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:00:46.829330 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:46.829648 18684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:46.829815 18684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "4d2a5209db464e49af94408b3c735186"
format_stamp: "Formatted at 2026-06-20 23:00:46 on dist-test-slave-rdl9"
server_key: "74e5289ef7a6881d5a5cc652d7f097c5"
server_key_iv: "23c1820fd6a39560ad4c301301978689"
server_key_version: "encryptionkey@0"
I20260620 23:00:46.829926 18684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:46.840871 18684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:46.843933 18684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:46.844105 18684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:46.848784 18684 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:44205
I20260620 23:00:46.848903 18751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:44205 every 8 connection(s)
I20260620 23:00:46.849265 18684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:46.850281 18752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:46.852907 18752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186: Bootstrap starting.
I20260620 23:00:46.853554 18752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:46.853978 18752 log.cc:824] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:46.855351 18752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186: No bootstrap required, opened a new log
I20260620 23:00:46.857077 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 18684
I20260620 23:00:46.857185 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:46.857580 1688 external_mini_cluster.cc:1428] Setting key 5ecf02b4dd8ca2377076ec78fddabdef
I20260620 23:00:46.857507 18752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2a5209db464e49af94408b3c735186" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44205 } }
I20260620 23:00:46.857694 18752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:46.857743 18752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2a5209db464e49af94408b3c735186, State: Initialized, Role: FOLLOWER
I20260620 23:00:46.857930 18752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2a5209db464e49af94408b3c735186" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44205 } }
I20260620 23:00:46.858039 18752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:46.858103 18752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:46.858170 18752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:46.859686 18752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2a5209db464e49af94408b3c735186" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44205 } }
I20260620 23:00:46.859818 18752 leader_election.cc:302] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d2a5209db464e49af94408b3c735186; no voters:
I20260620 23:00:46.860030 18752 leader_election.cc:288] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:46.860133 18757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:46.860374 18752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:46.860394 18757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [term 1 LEADER]: Becoming Leader. State: Replica: 4d2a5209db464e49af94408b3c735186, State: Running, Role: LEADER
I20260620 23:00:46.860737 18757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2a5209db464e49af94408b3c735186" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44205 } }
I20260620 23:00:46.861308 18759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d2a5209db464e49af94408b3c735186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2a5209db464e49af94408b3c735186" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44205 } } }
I20260620 23:00:46.861414 18759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:46.861673 18758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d2a5209db464e49af94408b3c735186. Latest consensus state: current_term: 1 leader_uuid: "4d2a5209db464e49af94408b3c735186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2a5209db464e49af94408b3c735186" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44205 } } }
I20260620 23:00:46.861814 18758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:46.862010 18766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:46.862654 18766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:46.865003 18766 catalog_manager.cc:1347] Generated new cluster ID: 085f2cd38b7640c0bddb537ad2f81e85
I20260620 23:00:46.865073 18766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:00:46 dist-test-slave-rdl9 krb5kdc[18671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:46.872174 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:46.858244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.1.166.62:44205)
0620 23:00:46.858517 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:46.858530 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:46.858579 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:00:46.858744 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:46.858753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:46.858848 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:46.859084 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:46.859821 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:46.861377 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:46.863003 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:46.863723 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:46.863889 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:46.867056 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:46.867062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:46.867063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:46.867081 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:46.869363 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:46.870123 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:46.870131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:46.870131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:46.870481 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:46.870903 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:46.870904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:46.870904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:46.871326 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:46.871559 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:46.871692 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 23:00:46.871882 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":139,"threads_started":1}
I20260620 23:00:46.873087 18766 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:46.873651 18766 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:46.882651 18766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186: Generated new TSK 0
I20260620 23:00:46.882936 18766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 23:00:46.911383 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44205
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:47.003996 18778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:47.004266 18778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:47.004295 18778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:47.005643 18778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:47.005714 18778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:47.005746 18778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 23:00:47.007551 18778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:47.007853 18778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:47.008127 18778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:47.014051 18783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:47.014057 18784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:47.014103 18778 server_base.cc:1034] running on GCE node
W20260620 23:00:47.014062 18786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:47.014725 18778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:47.015018 18778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:47.016211 18778 hybrid_clock.cc:648] HybridClock initialized: now 1781996447016192 us; error 37 us; skew 500 ppm
Jun 20 23:00:47 dist-test-slave-rdl9 krb5kdc[18671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996447, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:47.019568 18778 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 23:00:47.020146 18778 webserver.cc:460] Webserver started at http://127.1.166.1:44833/ using document root <none> and password file <none>
I20260620 23:00:47.020387 18778 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:47.020439 18778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:47.020584 18778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:47.021792 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "44eac936434b4367b249737f58029ea6"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "b230f0d8fc8fe14fe9792d6680ab9550"
server_key_iv: "79a3ef1e97cb0945ecee98d9c4dbba4d"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.022166 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "44eac936434b4367b249737f58029ea6"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "b230f0d8fc8fe14fe9792d6680ab9550"
server_key_iv: "79a3ef1e97cb0945ecee98d9c4dbba4d"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.023872 18778 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 23:00:47.024685 18793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.024935 18778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:47.025038 18778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "44eac936434b4367b249737f58029ea6"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "b230f0d8fc8fe14fe9792d6680ab9550"
server_key_iv: "79a3ef1e97cb0945ecee98d9c4dbba4d"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.025121 18778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:47.050146 18778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:47.052441 18778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:47.052590 18778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:47.052826 18778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:47.053164 18778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:47.053202 18778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.053243 18778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:47.053267 18778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.059903 18778 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:46693
I20260620 23:00:47.059994 18906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:46693 every 8 connection(s)
I20260620 23:00:47.060355 18778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:00:47 dist-test-slave-rdl9 krb5kdc[18671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996447, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:47.067747 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 18778
I20260620 23:00:47.067876 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 23:00:47.068096 1688 external_mini_cluster.cc:1428] Setting key 981adaf2d6a5cb65c353074caa81bf7a
I20260620 23:00:47.071966 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.061122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:53089 (local address 127.1.166.62:44205)
0620 23:00:47.061229 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.061232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.061412 (+ 180us) server_negotiation.cc:407] Connection header received
0620 23:00:47.061902 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.061903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.061921 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.062000 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:47.063136 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.063682 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.064717 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.064917 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.064975 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.067220 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:47.067222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:47.067222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:47.067230 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:47.069829 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:47.070593 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:47.070595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:47.070595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:47.070686 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:47.071111 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:47.071112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:47.071112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:47.071277 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:47.071363 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.071715 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.071849 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:00:47.072193 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.061101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:44205 (local address 127.1.166.1:53089)
0620 23:00:47.061258 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.061296 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:47.061785 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:47.062106 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:47.062110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:47.062800 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:47.063019 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.063025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.063781 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.063782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.064619 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.064622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.064995 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.064996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.065146 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.065693 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:47.065702 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:47.067114 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:47.069937 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:47.069958 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:47.069962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:47.070490 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:47.070759 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:47.070760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:47.070761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:47.070929 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:47.071357 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:47.071368 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:47.071560 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 23:00:47.071662 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:00:47.071928 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260620 23:00:47.072921 18907 heartbeater.cc:347] Connected to a master server at 127.1.166.62:44205
I20260620 23:00:47.073063 18907 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:47.073280 18907 heartbeater.cc:511] Master 127.1.166.62:44205 requested a full tablet report, sending...
I20260620 23:00:47.074115 18716 ts_manager.cc:194] Registered new tserver with Master: 44eac936434b4367b249737f58029ea6 (127.1.166.1:46693)
I20260620 23:00:47.075169 18716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:53089
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 23:00:47.097270 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44205
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:47.193389 18912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:47.193643 18912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:47.193673 18912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:47.195675 18912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:47.195804 18912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:47.195856 18912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 23:00:47.198210 18912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:47.198549 18912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:47.198851 18912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:47.205469 18917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:47.205456 18918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:47.205456 18920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:47.205899 18912 server_base.cc:1034] running on GCE node
I20260620 23:00:47.206147 18912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:47.206471 18912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:47.207679 18912 hybrid_clock.cc:648] HybridClock initialized: now 1781996447207654 us; error 35 us; skew 500 ppm
Jun 20 23:00:47 dist-test-slave-rdl9 krb5kdc[18671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996447, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:47.210875 18912 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 23:00:47.211527 18912 webserver.cc:460] Webserver started at http://127.1.166.2:42847/ using document root <none> and password file <none>
I20260620 23:00:47.211797 18912 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:47.211855 18912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:47.211997 18912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:47.213241 18912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "b6179d4eef2f404eb88cff5d3b24d7fd"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "46a2ae20d13c4e82140dcb77894de88a"
server_key_iv: "65900c732f058731c267fc4441538224"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.213991 18912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "b6179d4eef2f404eb88cff5d3b24d7fd"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "46a2ae20d13c4e82140dcb77894de88a"
server_key_iv: "65900c732f058731c267fc4441538224"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.215926 18912 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 23:00:47.217103 18927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.217444 18912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:47.217567 18912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "b6179d4eef2f404eb88cff5d3b24d7fd"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "46a2ae20d13c4e82140dcb77894de88a"
server_key_iv: "65900c732f058731c267fc4441538224"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.217675 18912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:47.237075 18912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:47.239828 18912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:47.240131 18912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:47.240460 18912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:47.240856 18912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:47.240900 18912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.240933 18912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:47.240952 18912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.247750 18912 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:36751
I20260620 23:00:47.247898 19040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:36751 every 8 connection(s)
I20260620 23:00:47.248456 18912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 23:00:47.253885 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 18912
I20260620 23:00:47.253984 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 23:00:47.254205 1688 external_mini_cluster.cc:1428] Setting key 6c88840afb1664a83e27e15da367c2a0
Jun 20 23:00:47 dist-test-slave-rdl9 krb5kdc[18671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996447, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:47.261287 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.249574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:39395 (local address 127.1.166.62:44205)
0620 23:00:47.249642 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.249645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.249892 (+ 247us) server_negotiation.cc:407] Connection header received
0620 23:00:47.250476 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.250478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.250492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.250563 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:47.252238 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.252786 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.254061 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.254466 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.254610 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.257907 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:47.257910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:47.257910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:47.257921 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:47.259478 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:47.260031 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:47.260033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:47.260033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:47.260100 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:47.260379 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:47.260380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:47.260381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:47.260555 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:47.260630 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.260977 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.261157 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:00:47.261556 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.249468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:44205 (local address 127.1.166.2:39395)
0620 23:00:47.249687 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.249740 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:47.250383 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:47.250761 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:47.250768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:47.251629 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:47.252003 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.252029 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.252930 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.252931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.253894 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.253898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.254641 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.254642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.254900 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.255787 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:47.255802 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:47.257736 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:47.259604 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:47.259617 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:47.259620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:47.259918 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:47.260165 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:47.260166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:47.260166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:47.260307 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:47.260626 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:47.260636 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:47.260831 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 23:00:47.260948 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:00:47.261183 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
I20260620 23:00:47.262398 19041 heartbeater.cc:347] Connected to a master server at 127.1.166.62:44205
I20260620 23:00:47.262529 19041 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:47.262759 19041 heartbeater.cc:511] Master 127.1.166.62:44205 requested a full tablet report, sending...
I20260620 23:00:47.263202 18716 ts_manager.cc:194] Registered new tserver with Master: b6179d4eef2f404eb88cff5d3b24d7fd (127.1.166.2:36751)
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
I20260620 23:00:47.263612 18716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:39395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 23:00:47.283263 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44205
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:47.372191 19046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:47.372458 19046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:47.372486 19046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:47.373700 19046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:47.373746 19046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:47.373773 19046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 23:00:47.375671 19046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:47.376216 19046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:47.376529 19046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:47.382059 19054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:47.382232 19051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:47.382091 19052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:47.382313 19046 server_base.cc:1034] running on GCE node
I20260620 23:00:47.382570 19046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:47.382849 19046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:47.384033 19046 hybrid_clock.cc:648] HybridClock initialized: now 1781996447384019 us; error 31 us; skew 500 ppm
Jun 20 23:00:47 dist-test-slave-rdl9 krb5kdc[18671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996447, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:47.387156 19046 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 23:00:47.387728 19046 webserver.cc:460] Webserver started at http://127.1.166.3:38441/ using document root <none> and password file <none>
I20260620 23:00:47.387967 19046 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:47.388038 19046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:47.388190 19046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:47.389384 19046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "15afb87db8c44d3bb16e67a0a55bca8d"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "1e57e40bef600ba48e9748be427b214d"
server_key_iv: "ccaf2b720cd5f92f6889e18dbab6e8da"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.389802 19046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "15afb87db8c44d3bb16e67a0a55bca8d"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "1e57e40bef600ba48e9748be427b214d"
server_key_iv: "ccaf2b720cd5f92f6889e18dbab6e8da"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.391671 19046 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 23:00:47.392926 19061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.393141 19046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:47.393250 19046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "15afb87db8c44d3bb16e67a0a55bca8d"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "1e57e40bef600ba48e9748be427b214d"
server_key_iv: "ccaf2b720cd5f92f6889e18dbab6e8da"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.393347 19046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:47.420019 19046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:47.422676 19046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:47.422835 19046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:47.423077 19046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:47.423399 19046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:47.423434 19046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.423465 19046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:47.423482 19046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.429447 19046 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:36347
I20260620 23:00:47.429574 19174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:36347 every 8 connection(s)
I20260620 23:00:47.430131 19046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:00:47 dist-test-slave-rdl9 krb5kdc[18671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996447, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:47.439165 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 19046
I20260620 23:00:47.439277 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 23:00:47.439435 1688 external_mini_cluster.cc:1428] Setting key 347dce21c54a218ea4bd629468510b67
I20260620 23:00:47.442773 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.431230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:36605 (local address 127.1.166.62:44205)
0620 23:00:47.431354 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.431357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.431602 (+ 245us) server_negotiation.cc:407] Connection header received
0620 23:00:47.432131 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.432133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.432147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.432238 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:47.433398 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.433988 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.435392 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.435641 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.435735 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.438973 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:47.438975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:47.438975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:47.438986 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:47.440874 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:47.441441 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:47.441443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:47.441444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:47.441545 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:47.441941 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:47.441943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:47.441943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:47.442113 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:47.442189 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.442593 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.442683 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:00:47.442973 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.431119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:44205 (local address 127.1.166.3:36605)
0620 23:00:47.431435 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.431471 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:47.431983 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:47.432318 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:47.432322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:47.432971 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:47.433200 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.433216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.434161 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.434163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.435233 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.435237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.435770 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.435772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.435999 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.436629 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:47.436641 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:47.438775 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:47.440987 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:47.440998 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:47.441000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:47.441297 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:47.441668 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:47.441669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:47.441669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:47.441817 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:47.442212 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:47.442218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:47.442426 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 23:00:47.442535 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:00:47.442713 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260620 23:00:47.443499 19175 heartbeater.cc:347] Connected to a master server at 127.1.166.62:44205
I20260620 23:00:47.443612 19175 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:47.443811 19175 heartbeater.cc:511] Master 127.1.166.62:44205 requested a full tablet report, sending...
I20260620 23:00:47.444242 18716 ts_manager.cc:194] Registered new tserver with Master: 15afb87db8c44d3bb16e67a0a55bca8d (127.1.166.3:36347)
I20260620 23:00:47.444659 18716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:36605
I20260620 23:00:47.452525 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:47.460145 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.453902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.1.166.62:44205)
0620 23:00:47.453956 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.453958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.454095 (+ 137us) server_negotiation.cc:407] Connection header received
0620 23:00:47.454159 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.454160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.454171 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.454225 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:47.454832 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.455327 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.456034 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.456217 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.456272 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.457636 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:47.457637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:47.457638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:47.457645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:47.459068 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:47.459397 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:47.459398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:47.459398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:47.459448 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:47.459618 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:47.459619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:47.459620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:47.459771 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:47.459862 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.459968 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.460054 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:00:47.462282 18716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:00:47.464049 18716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:47.471702 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.466830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.1.166.1:46693)
0620 23:00:47.467066 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.467074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.467093 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:00:47.467152 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.467155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.467174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.467351 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:47.467716 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.468855 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.470249 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.471062 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.471166 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.471240 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.471527 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.471584 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260620 23:00:47.471743 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.466565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:36347 (local address 127.0.0.1:58602)
0620 23:00:47.467027 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.467037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:47.467075 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:47.467409 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:47.467413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:47.467416 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:47.467506 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.467508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.468969 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.468970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.470407 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.470411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.471308 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.471309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.471368 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.471389 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:00:47.471491 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:00:47.471606 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":93,"thread_start_us":84,"threads_started":1}
I20260620 23:00:47.471743 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.466565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46693 (local address 127.0.0.1:48292)
0620 23:00:47.466834 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.466876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:47.466944 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:47.467409 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:47.467413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:47.467418 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:47.467601 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.467603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.468996 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.468998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.470155 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.470159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.471204 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.471206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.471365 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.471389 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:00:47.471491 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:00:47.471604 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":221,"thread_start_us":61,"threads_started":1}
I20260620 23:00:47.472869 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.466612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:36751 (local address 127.0.0.1:41746)
0620 23:00:47.467227 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.467239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:47.467311 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:47.467513 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:47.467515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:47.467517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:47.467709 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.467712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.468888 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.468889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.470408 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.470411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.472576 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.472577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.472655 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.472660 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:47.472727 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:00:47.472794 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":116,"thread_start_us":146,"threads_started":1}
I20260620 23:00:47.473086 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.467005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.1.166.2:36751)
0620 23:00:47.467246 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.467250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.467300 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:00:47.467420 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.467424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.467439 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.467510 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:47.467798 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.468733 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.471351 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.472265 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.472436 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.472527 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.472899 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.472982 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":116,"threads_started":1}
I20260620 23:00:47.473049 18841 tablet_service.cc:1467] Processing CreateTablet for tablet 2110881c02d74f07a2cb0c21859a05a1 (DEFAULT_TABLE table=test-table [id=a63f1591c2db49e481327b48df3cdba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:47.473488 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2110881c02d74f07a2cb0c21859a05a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:47.474800 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.466750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.1.166.3:36347)
0620 23:00:47.467210 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.467216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.467227 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:00:47.467270 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.467272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.467289 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.467583 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:47.467672 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.468876 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.470498 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.471192 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.471260 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.471305 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.474606 (+ 3301us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.474683 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":103,"threads_started":1}
I20260620 23:00:47.475236 18975 tablet_service.cc:1467] Processing CreateTablet for tablet 2110881c02d74f07a2cb0c21859a05a1 (DEFAULT_TABLE table=test-table [id=a63f1591c2db49e481327b48df3cdba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:47.475596 18975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2110881c02d74f07a2cb0c21859a05a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:47.475556 19109 tablet_service.cc:1467] Processing CreateTablet for tablet 2110881c02d74f07a2cb0c21859a05a1 (DEFAULT_TABLE table=test-table [id=a63f1591c2db49e481327b48df3cdba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:47.475871 19109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2110881c02d74f07a2cb0c21859a05a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:47.477543 19192 tablet_bootstrap.cc:492] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6: Bootstrap starting.
I20260620 23:00:47.478458 19192 tablet_bootstrap.cc:654] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:47.479089 19192 log.cc:824] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:47.479296 19193 tablet_bootstrap.cc:492] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: Bootstrap starting.
I20260620 23:00:47.479630 19194 tablet_bootstrap.cc:492] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Bootstrap starting.
I20260620 23:00:47.480343 19193 tablet_bootstrap.cc:654] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:47.480466 19194 tablet_bootstrap.cc:654] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:47.480997 19193 log.cc:824] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:47.481010 19194 log.cc:824] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:47.481865 19192 tablet_bootstrap.cc:492] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6: No bootstrap required, opened a new log
I20260620 23:00:47.481874 19193 tablet_bootstrap.cc:492] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: No bootstrap required, opened a new log
I20260620 23:00:47.481949 19193 ts_tablet_manager.cc:1397] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:00:47.481949 19192 ts_tablet_manager.cc:1397] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 23:00:47.482662 19194 tablet_bootstrap.cc:492] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: No bootstrap required, opened a new log
I20260620 23:00:47.482736 19194 ts_tablet_manager.cc:1397] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:00:47.484032 19192 raft_consensus.cc:348] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.484228 19192 raft_consensus.cc:374] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:47.484266 19192 raft_consensus.cc:729] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44eac936434b4367b249737f58029ea6, State: Initialized, Role: FOLLOWER
I20260620 23:00:47.484395 19192 consensus_queue.cc:260] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.484326 19194 raft_consensus.cc:348] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.484318 19193 raft_consensus.cc:348] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.484481 19194 raft_consensus.cc:374] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:47.484494 19193 raft_consensus.cc:374] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:47.484522 19194 raft_consensus.cc:729] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6179d4eef2f404eb88cff5d3b24d7fd, State: Initialized, Role: FOLLOWER
I20260620 23:00:47.484527 19193 raft_consensus.cc:729] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15afb87db8c44d3bb16e67a0a55bca8d, State: Initialized, Role: FOLLOWER
I20260620 23:00:47.484642 19194 consensus_queue.cc:260] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.484647 19193 consensus_queue.cc:260] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.484695 19192 ts_tablet_manager.cc:1428] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 23:00:47.484833 18907 heartbeater.cc:503] Master 127.1.166.62:44205 was elected leader, sending a full tablet report...
I20260620 23:00:47.484905 19194 ts_tablet_manager.cc:1428] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:47.484907 19193 ts_tablet_manager.cc:1428] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:47.485008 19041 heartbeater.cc:503] Master 127.1.166.62:44205 was elected leader, sending a full tablet report...
I20260620 23:00:47.485008 19175 heartbeater.cc:503] Master 127.1.166.62:44205 was elected leader, sending a full tablet report...
W20260620 23:00:47.499167 19042 tablet.cc:2369] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:00:47.561342 18908 tablet.cc:2369] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:00:47.580996 19199 raft_consensus.cc:482] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:47.581183 19199 raft_consensus.cc:504] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.581637 19199 leader_election.cc:288] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15afb87db8c44d3bb16e67a0a55bca8d (127.1.166.3:36347), 44eac936434b4367b249737f58029ea6 (127.1.166.1:46693)
I20260620 23:00:47.584847 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.581909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:36347 (local address 127.1.166.2:45887)
0620 23:00:47.581991 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.582006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:47.582058 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:47.582407 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:47.582408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:47.582411 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:47.582502 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.582505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.583202 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.583203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.584003 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.584005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.584569 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.584570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.584650 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.584653 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:47.584697 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 23:00:47.584741 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":5}
I20260620 23:00:47.584776 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.582036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:35829 (local address 127.1.166.1:46693)
0620 23:00:47.582096 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.582098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.582245 (+ 147us) server_negotiation.cc:407] Connection header received
0620 23:00:47.582450 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.582451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.582464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.582531 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:47.582700 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.583149 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.584088 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.584450 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.584511 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.584522 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.584655 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.584705 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 23:00:47.584952 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.581956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46693 (local address 127.1.166.2:35829)
0620 23:00:47.582168 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.582176 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:47.582234 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:47.582548 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:47.582549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:47.582552 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:47.582630 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.582631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.583234 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.583235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.583996 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:47.583999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.584523 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.584524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:47.584597 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.584602 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:47.584650 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:00:47.584693 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":123,"threads_started":1}
I20260620 23:00:47.585073 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.582077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:45887 (local address 127.1.166.3:36347)
0620 23:00:47.582141 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.582143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.582153 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:00:47.582191 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.582192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.582205 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.582430 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:47.582576 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.583115 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.584099 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.584493 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.584555 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.584568 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.584798 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.584989 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:00:47.585449 19129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2110881c02d74f07a2cb0c21859a05a1" candidate_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" is_pre_election: true
I20260620 23:00:47.585347 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2110881c02d74f07a2cb0c21859a05a1" candidate_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44eac936434b4367b249737f58029ea6" is_pre_election: true
I20260620 23:00:47.585644 19129 raft_consensus.cc:2454] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6179d4eef2f404eb88cff5d3b24d7fd in term 0.
I20260620 23:00:47.585645 18861 raft_consensus.cc:2454] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6179d4eef2f404eb88cff5d3b24d7fd in term 0.
I20260620 23:00:47.585952 18928 leader_election.cc:302] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb87db8c44d3bb16e67a0a55bca8d, b6179d4eef2f404eb88cff5d3b24d7fd; no voters:
I20260620 23:00:47.586153 19199 raft_consensus.cc:2781] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:47.586217 19199 raft_consensus.cc:482] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:47.586241 19199 raft_consensus.cc:3037] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:47.587255 19199 raft_consensus.cc:504] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.587417 19199 leader_election.cc:288] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [CANDIDATE]: Term 1 election: Requested vote from peers 15afb87db8c44d3bb16e67a0a55bca8d (127.1.166.3:36347), 44eac936434b4367b249737f58029ea6 (127.1.166.1:46693)
I20260620 23:00:47.587634 19129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2110881c02d74f07a2cb0c21859a05a1" candidate_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15afb87db8c44d3bb16e67a0a55bca8d"
I20260620 23:00:47.587682 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2110881c02d74f07a2cb0c21859a05a1" candidate_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44eac936434b4367b249737f58029ea6"
I20260620 23:00:47.587766 18861 raft_consensus.cc:3037] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:47.587766 19129 raft_consensus.cc:3037] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:47.588737 19129 raft_consensus.cc:2454] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6179d4eef2f404eb88cff5d3b24d7fd in term 1.
I20260620 23:00:47.588749 18861 raft_consensus.cc:2454] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6179d4eef2f404eb88cff5d3b24d7fd in term 1.
I20260620 23:00:47.588925 18928 leader_election.cc:302] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15afb87db8c44d3bb16e67a0a55bca8d, b6179d4eef2f404eb88cff5d3b24d7fd; no voters:
I20260620 23:00:47.589027 19199 raft_consensus.cc:2781] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:47.589218 19199 raft_consensus.cc:686] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 1 LEADER]: Becoming Leader. State: Replica: b6179d4eef2f404eb88cff5d3b24d7fd, State: Running, Role: LEADER
I20260620 23:00:47.589318 19199 consensus_queue.cc:237] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } }
I20260620 23:00:47.590767 18715 catalog_manager.cc:5495] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd reported cstate change: term changed from 0 to 1, leader changed from <none> to b6179d4eef2f404eb88cff5d3b24d7fd (127.1.166.2). New cstate: current_term: 1 leader_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6179d4eef2f404eb88cff5d3b24d7fd" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 36751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:47.619889 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.617358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.1.166.2:36751)
0620 23:00:47.617421 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.617424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.617434 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:00:47.617480 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.617481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.617504 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.617571 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:00:47.617730 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.618209 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.619055 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.619250 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.619310 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.619484 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:00:47.619603 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:00:47.619692 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.619742 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.619767 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:00:47.622285 18861 raft_consensus.cc:1261] T 2110881c02d74f07a2cb0c21859a05a1 P 44eac936434b4367b249737f58029ea6 [term 1 FOLLOWER]: Refusing update from remote peer b6179d4eef2f404eb88cff5d3b24d7fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:47.622285 19129 raft_consensus.cc:1261] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 1 FOLLOWER]: Refusing update from remote peer b6179d4eef2f404eb88cff5d3b24d7fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:47.622731 19199 consensus_queue.cc:1035] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "15afb87db8c44d3bb16e67a0a55bca8d" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:47.622933 19202 consensus_queue.cc:1035] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44eac936434b4367b249737f58029ea6" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:47.625828 19205 mvcc.cc:205] Tried to move back new op lower bound from 7299057449457586176 to 7299057449327108096. Current Snapshot: MvccSnapshot[applied={T|T < 7299057449457586176}]
I20260620 23:00:47.626487 19206 mvcc.cc:205] Tried to move back new op lower bound from 7299057449457586176 to 7299057449327108096. Current Snapshot: MvccSnapshot[applied={T|T < 7299057449457586176}]
I20260620 23:00:47.629390 18714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:47.629565 18714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:47.630767 18714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d2a5209db464e49af94408b3c735186: Sending DeleteTablet for 3 replicas of tablet 2110881c02d74f07a2cb0c21859a05a1
I20260620 23:00:47.631506 18975 tablet_service.cc:1514] Processing DeleteTablet for tablet 2110881c02d74f07a2cb0c21859a05a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:47 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:41746
I20260620 23:00:47.631554 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet 2110881c02d74f07a2cb0c21859a05a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:47 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:48292
I20260620 23:00:47.631606 19109 tablet_service.cc:1514] Processing DeleteTablet for tablet 2110881c02d74f07a2cb0c21859a05a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:47 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:58602
I20260620 23:00:47.631940 19216 tablet_replica.cc:330] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: stopping tablet replica
I20260620 23:00:47.632079 19216 raft_consensus.cc:2229] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:00:47.632248 19216 raft_consensus.cc:2258] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:47.632290 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 18778
I20260620 23:00:47.633998 19216 ts_tablet_manager.cc:1905] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:47.636515 19217 tablet_replica.cc:330] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: stopping tablet replica
I20260620 23:00:47.636550 19216 ts_tablet_manager.cc:1918] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:47.636632 19216 log.cc:1196] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/2110881c02d74f07a2cb0c21859a05a1
I20260620 23:00:47.636704 19217 raft_consensus.cc:2229] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:47.636859 19217 raft_consensus.cc:2258] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:47.637419 19217 ts_tablet_manager.cc:1905] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:47.639142 19217 ts_tablet_manager.cc:1918] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:47.639246 19217 log.cc:1196] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/2110881c02d74f07a2cb0c21859a05a1
I20260620 23:00:47.639487 19217 ts_tablet_manager.cc:1939] T 2110881c02d74f07a2cb0c21859a05a1 P 15afb87db8c44d3bb16e67a0a55bca8d: Deleting consensus metadata
I20260620 23:00:47.640012 18700 catalog_manager.cc:4841] TS 15afb87db8c44d3bb16e67a0a55bca8d (127.1.166.3:36347): tablet 2110881c02d74f07a2cb0c21859a05a1 (table test-table [id=a63f1591c2db49e481327b48df3cdba3]) successfully deleted
W20260620 23:00:47.640914 18702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:46693 (error 108)
I20260620 23:00:47.641079 19216 ts_tablet_manager.cc:1939] T 2110881c02d74f07a2cb0c21859a05a1 P b6179d4eef2f404eb88cff5d3b24d7fd: Deleting consensus metadata
I20260620 23:00:47.641585 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.641440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46693 (local address 127.0.0.1:48298)
0620 23:00:47.641499 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:47.641544 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:00:47.641594 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 18912
I20260620 23:00:47.642081 18701 catalog_manager.cc:4841] TS b6179d4eef2f404eb88cff5d3b24d7fd (127.1.166.2:36751): tablet 2110881c02d74f07a2cb0c21859a05a1 (table test-table [id=a63f1591c2db49e481327b48df3cdba3]) successfully deleted
W20260620 23:00:47.642385 18702 catalog_manager.cc:4584] TS 44eac936434b4367b249737f58029ea6 (127.1.166.1:46693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2110881c02d74f07a2cb0c21859a05a1: Network error: Client connection negotiation failed: client connection to 127.1.166.1:46693: connect: Connection refused (error 111)
I20260620 23:00:47.646270 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 19046
I20260620 23:00:47.651795 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 18684
2026-06-20T23:00:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2070 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T23:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:47Z Disabled control of system clock
I20260620 23:00:47.683104 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:41149
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:00:47.776906 19222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:47.777169 19222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:47.777192 19222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:47.778463 19222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:47.778508 19222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:47.778519 19222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:47.778532 19222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:47.778543 19222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:47.779994 19222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:41149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:47.780215 19222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:47.780488 19222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:00:47.785708 19222 server_base.cc:1034] running on GCE node
W20260620 23:00:47.785677 19230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:47.785668 19228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:47.785653 19227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:47.786090 19222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:47.786450 19222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:47.787640 19222 hybrid_clock.cc:648] HybridClock initialized: now 1781996447787632 us; error 46 us; skew 500 ppm
I20260620 23:00:47.789062 19222 webserver.cc:460] Webserver started at http://127.1.166.62:45537/ using document root <none> and password file <none>
I20260620 23:00:47.789286 19222 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:47.789338 19222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:47.789431 19222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:47.790570 19222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "6a00e9f80e774aa5b282641f1ceed3b2"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "ec351eaab84c1b5d7c39ce43cb2e08b9"
server_key_iv: "716ab54ca8690d44818f2517ec4c60d8"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.790882 19222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "6a00e9f80e774aa5b282641f1ceed3b2"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "ec351eaab84c1b5d7c39ce43cb2e08b9"
server_key_iv: "716ab54ca8690d44818f2517ec4c60d8"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.792104 19222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:47.792822 19236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.792994 19222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:00:47.793078 19222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "6a00e9f80e774aa5b282641f1ceed3b2"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "ec351eaab84c1b5d7c39ce43cb2e08b9"
server_key_iv: "716ab54ca8690d44818f2517ec4c60d8"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.793149 19222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:47.811003 19222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:47.811311 19222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:47.811419 19222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:47.816303 19222 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:41149
I20260620 23:00:47.816416 19288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:41149 every 8 connection(s)
I20260620 23:00:47.816834 19222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:47.818032 19289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:47.818908 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 19222
I20260620 23:00:47.819027 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:47.819242 1688 external_mini_cluster.cc:1428] Setting key c61f3480926631775613e469e1042293
I20260620 23:00:47.822108 19289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2: Bootstrap starting.
I20260620 23:00:47.822956 19289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:47.823269 19289 log.cc:824] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:47.824347 19289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2: No bootstrap required, opened a new log
I20260620 23:00:47.825994 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:47.819838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.1.166.62:41149)
0620 23:00:47.820159 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:47.820173 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:47.820213 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:00:47.820902 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:47.820910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:47.820990 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:47.821149 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:47.821411 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.822504 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.823416 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:47.823937 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:47.824068 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:47.825027 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:47.825033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:47.825033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:47.825291 (+ 258us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:47.825341 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:47.825359 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:47.825445 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:47.825626 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 23:00:47.825734 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":121,"threads_started":1}
I20260620 23:00:47.827075 19289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a00e9f80e774aa5b282641f1ceed3b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41149 } }
I20260620 23:00:47.827248 19289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:47.827272 19289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a00e9f80e774aa5b282641f1ceed3b2, State: Initialized, Role: FOLLOWER
I20260620 23:00:47.827395 19289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a00e9f80e774aa5b282641f1ceed3b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41149 } }
I20260620 23:00:47.827481 19289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:47.827512 19289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:47.827543 19289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:47.828294 19289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a00e9f80e774aa5b282641f1ceed3b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41149 } }
I20260620 23:00:47.828399 19289 leader_election.cc:302] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a00e9f80e774aa5b282641f1ceed3b2; no voters:
I20260620 23:00:47.828604 19289 leader_election.cc:288] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:47.828729 19294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:47.828894 19294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 6a00e9f80e774aa5b282641f1ceed3b2, State: Running, Role: LEADER
I20260620 23:00:47.828959 19289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:47.828997 19294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a00e9f80e774aa5b282641f1ceed3b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41149 } }
I20260620 23:00:47.829394 19296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a00e9f80e774aa5b282641f1ceed3b2. Latest consensus state: current_term: 1 leader_uuid: "6a00e9f80e774aa5b282641f1ceed3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a00e9f80e774aa5b282641f1ceed3b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41149 } } }
I20260620 23:00:47.829396 19295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a00e9f80e774aa5b282641f1ceed3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a00e9f80e774aa5b282641f1ceed3b2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41149 } } }
I20260620 23:00:47.829497 19296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:47.829553 19295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2 [sys.catalog]: This master's current role is: LEADER
W20260620 23:00:47.830938 19310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:00:47.830991 19310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:00:47.831050 19302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:47.831265 19302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:47.833359 19302 catalog_manager.cc:1347] Generated new cluster ID: 830acfc4b3e947cabbf79aaa16a7345b
I20260620 23:00:47.833423 19302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:47.844336 19302 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:47.845027 19302 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:47.853981 19302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2: Generated new TSK 0
I20260620 23:00:47.854259 19302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:47.855772 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41149
--builtin_ntp_servers=127.1.166.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:00:47.949503 19313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:47.949762 19313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:47.949795 19313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:47.951974 19313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:47.952054 19313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:47.952095 19313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 23:00:47.953962 19313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:47.954275 19313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:47.954658 19313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:47.960595 19319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:47.960680 19313 server_base.cc:1034] running on GCE node
W20260620 23:00:47.960525 19318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:47.960505 19321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:47.961097 19313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:47.961383 19313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:47.962546 19313 hybrid_clock.cc:648] HybridClock initialized: now 1781996447962533 us; error 28 us; skew 500 ppm
I20260620 23:00:47.963987 19313 webserver.cc:460] Webserver started at http://127.1.166.1:45317/ using document root <none> and password file <none>
I20260620 23:00:47.964242 19313 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:47.964288 19313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:47.964416 19313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:47.965554 19313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "338ae55f38a3fb5e4169068b41684970"
server_key_iv: "0cd6682f52ddc5d181b303e5430e9859"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.965904 19313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "338ae55f38a3fb5e4169068b41684970"
server_key_iv: "0cd6682f52ddc5d181b303e5430e9859"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.967260 19313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:00:47.968351 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.968555 19313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:47.968667 19313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc"
format_stamp: "Formatted at 2026-06-20 23:00:47 on dist-test-slave-rdl9"
server_key: "338ae55f38a3fb5e4169068b41684970"
server_key_iv: "0cd6682f52ddc5d181b303e5430e9859"
server_key_version: "encryptionkey@0"
I20260620 23:00:47.968779 19313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:47.990252 19313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:47.990756 19313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:47.990892 19313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:47.991199 19313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:47.991609 19313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:47.991647 19313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.991680 19313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:47.991704 19313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:47.998981 19313 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:46747
I20260620 23:00:47.999053 19440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:46747 every 8 connection(s)
I20260620 23:00:47.999433 19313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 23:00:48.001689 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 19313
I20260620 23:00:48.001791 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 23:00:48.001994 1688 external_mini_cluster.cc:1428] Setting key 19a0cf751289d1746b432ca16b42635a
I20260620 23:00:48.002914 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41149
--builtin_ntp_servers=127.1.166.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:00:48.005889 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.000276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:55123 (local address 127.1.166.62:41149)
0620 23:00:48.000367 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.000369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.000537 (+ 168us) server_negotiation.cc:407] Connection header received
0620 23:00:48.001090 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.001091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.001108 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.001178 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:48.002102 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.002699 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.004253 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.004474 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.004556 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.005587 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:48.005589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:48.005590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:48.005595 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:48.005614 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:48.005618 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:48.005672 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.005779 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.005820 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:00:48.006207 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.000229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41149 (local address 127.1.166.1:55123)
0620 23:00:48.000412 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.000456 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:48.000960 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:48.001316 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:48.001321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:48.001330 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:48.001969 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.001975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.002793 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.002795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.004103 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.004108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.004572 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.004572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.004759 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.005400 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:48.005401 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:48.005443 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:48.005475 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:48.005684 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:48.005688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:48.005691 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:48.005759 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:00:48.005888 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260620 23:00:48.007248 19441 heartbeater.cc:347] Connected to a master server at 127.1.166.62:41149
I20260620 23:00:48.007393 19441 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:48.007642 19441 heartbeater.cc:511] Master 127.1.166.62:41149 requested a full tablet report, sending...
I20260620 23:00:48.008324 19253 ts_manager.cc:194] Registered new tserver with Master: 8a1a3d6119b84a1eb3f58e912a4e9cfc (127.1.166.1:46747)
I20260620 23:00:48.009217 19253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.1:55123
W20260620 23:00:48.093510 19444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:48.093765 19444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:48.093798 19444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:48.095654 19444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:48.095768 19444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:48.095821 19444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 23:00:48.097868 19444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:48.098181 19444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:48.098599 19444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:48.104612 19449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:48.104596 19452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:48.104590 19450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:48.105020 19444 server_base.cc:1034] running on GCE node
I20260620 23:00:48.105288 19444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:48.105554 19444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:48.106717 19444 hybrid_clock.cc:648] HybridClock initialized: now 1781996448106699 us; error 33 us; skew 500 ppm
I20260620 23:00:48.108497 19444 webserver.cc:460] Webserver started at http://127.1.166.2:45191/ using document root <none> and password file <none>
I20260620 23:00:48.108779 19444 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:48.108834 19444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:48.108964 19444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:48.110574 19444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "b58681e133604e2cbd7d13dadd5080e4"
format_stamp: "Formatted at 2026-06-20 23:00:48 on dist-test-slave-rdl9"
server_key: "e328a8b16a3719c6b01d9d9715da4c92"
server_key_iv: "8da4a059a9392abe50ee8d7c1b397a61"
server_key_version: "encryptionkey@0"
I20260620 23:00:48.111009 19444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "b58681e133604e2cbd7d13dadd5080e4"
format_stamp: "Formatted at 2026-06-20 23:00:48 on dist-test-slave-rdl9"
server_key: "e328a8b16a3719c6b01d9d9715da4c92"
server_key_iv: "8da4a059a9392abe50ee8d7c1b397a61"
server_key_version: "encryptionkey@0"
I20260620 23:00:48.112591 19444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:00:48.113673 19458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:48.113978 19444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:48.114094 19444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "b58681e133604e2cbd7d13dadd5080e4"
format_stamp: "Formatted at 2026-06-20 23:00:48 on dist-test-slave-rdl9"
server_key: "e328a8b16a3719c6b01d9d9715da4c92"
server_key_iv: "8da4a059a9392abe50ee8d7c1b397a61"
server_key_version: "encryptionkey@0"
I20260620 23:00:48.114195 19444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:48.123512 19444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:48.123870 19444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:48.124010 19444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:48.124257 19444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:48.124624 19444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:48.124665 19444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:48.124697 19444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:48.124724 19444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:48.132162 19444 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:45135
I20260620 23:00:48.132308 19571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:45135 every 8 connection(s)
I20260620 23:00:48.132663 19444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 23:00:48.139019 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 19444
I20260620 23:00:48.139122 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.133656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:35673 (local address 127.1.166.62:41149)
0620 23:00:48.133737 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.133740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.134067 (+ 327us) server_negotiation.cc:407] Connection header received
0620 23:00:48.134616 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.134617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.134630 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.134681 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:48.135460 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.135990 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.137083 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.137416 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.137495 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.138505 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:48.138506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:48.138507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:48.138512 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:48.138529 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:48.138534 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:48.138585 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.138876 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.138939 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:00:48.139160 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 23:00:48.139308 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.133643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41149 (local address 127.1.166.2:35673)
0620 23:00:48.133895 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.133934 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:48.134507 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:48.134768 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:48.134771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:48.134781 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:48.135333 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.135338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.136098 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.136100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.136957 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.136960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.137535 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.137536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.137679 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.138281 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:48.138281 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:48.138314 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:48.138384 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:48.138655 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:48.138666 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:48.138672 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:00:48.138838 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 23:00:48.139012 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
I20260620 23:00:48.139561 1688 external_mini_cluster.cc:1428] Setting key c902829b401d33ec9a37b7bd3ff066b8
I20260620 23:00:48.139976 19572 heartbeater.cc:347] Connected to a master server at 127.1.166.62:41149
I20260620 23:00:48.140112 19572 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:48.140386 19572 heartbeater.cc:511] Master 127.1.166.62:41149 requested a full tablet report, sending...
I20260620 23:00:48.140764 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41149
--builtin_ntp_servers=127.1.166.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:00:48.140851 19253 ts_manager.cc:194] Registered new tserver with Master: b58681e133604e2cbd7d13dadd5080e4 (127.1.166.2:45135)
I20260620 23:00:48.141263 19253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.2:35673
W20260620 23:00:48.234632 19575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:48.234928 19575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:48.234956 19575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:48.236479 19575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:48.236552 19575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:48.236583 19575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 23:00:48.238596 19575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:48.238968 19575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:48.239362 19575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:48.245857 19581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:48.245786 19583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:48.245786 19580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:48.246603 19575 server_base.cc:1034] running on GCE node
I20260620 23:00:48.246915 19575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:48.247210 19575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:48.248410 19575 hybrid_clock.cc:648] HybridClock initialized: now 1781996448248381 us; error 43 us; skew 500 ppm
I20260620 23:00:48.250283 19575 webserver.cc:460] Webserver started at http://127.1.166.3:39873/ using document root <none> and password file <none>
I20260620 23:00:48.250630 19575 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:48.250691 19575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:48.250842 19575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:48.252420 19575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "d55e029a4c054add86536445ef5d1b43"
format_stamp: "Formatted at 2026-06-20 23:00:48 on dist-test-slave-rdl9"
server_key: "b47c5a082c3d1452b0e2213f6d5861a6"
server_key_iv: "faac1228e7d528e9f88e5310898270c1"
server_key_version: "encryptionkey@0"
I20260620 23:00:48.252939 19575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "d55e029a4c054add86536445ef5d1b43"
format_stamp: "Formatted at 2026-06-20 23:00:48 on dist-test-slave-rdl9"
server_key: "b47c5a082c3d1452b0e2213f6d5861a6"
server_key_iv: "faac1228e7d528e9f88e5310898270c1"
server_key_version: "encryptionkey@0"
I20260620 23:00:48.254492 19575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 23:00:48.255476 19589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:48.255828 19575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:48.255957 19575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "d55e029a4c054add86536445ef5d1b43"
format_stamp: "Formatted at 2026-06-20 23:00:48 on dist-test-slave-rdl9"
server_key: "b47c5a082c3d1452b0e2213f6d5861a6"
server_key_iv: "faac1228e7d528e9f88e5310898270c1"
server_key_version: "encryptionkey@0"
I20260620 23:00:48.256048 19575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:48.264907 19575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:48.265240 19575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:48.265367 19575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:48.265578 19575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:48.265936 19575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:48.265973 19575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:48.266003 19575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:48.266019 19575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:48.273136 19575 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:35149
I20260620 23:00:48.273219 19702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:35149 every 8 connection(s)
I20260620 23:00:48.273701 19575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 23:00:48.277289 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 19575
I20260620 23:00:48.277403 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 23:00:48.277603 1688 external_mini_cluster.cc:1428] Setting key 9e56702206173e789ac80b1547724b8c
I20260620 23:00:48.279834 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.274789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:49989 (local address 127.1.166.62:41149)
0620 23:00:48.274849 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.274851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.275105 (+ 254us) server_negotiation.cc:407] Connection header received
0620 23:00:48.275684 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.275686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.275700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.275756 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:48.276537 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.277042 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.278279 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.278582 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.278645 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.279556 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:48.279557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:48.279557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:48.279563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:48.279580 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:48.279585 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:48.279631 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.279712 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.279742 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:00:48.280103 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.274719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41149 (local address 127.1.166.3:49989)
0620 23:00:48.274977 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.275016 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:48.275572 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:48.275841 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:48.275845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:48.275855 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:48.276432 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.276438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.277165 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.277166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.278210 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.278214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.278668 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.278670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.278826 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.279370 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:48.279370 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:48.279407 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:48.279438 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:48.279641 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:48.279647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:48.279650 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:48.279716 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:00:48.279855 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":126,"threads_started":1}
I20260620 23:00:48.280622 19703 heartbeater.cc:347] Connected to a master server at 127.1.166.62:41149
I20260620 23:00:48.280730 19703 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:48.280941 19703 heartbeater.cc:511] Master 127.1.166.62:41149 requested a full tablet report, sending...
I20260620 23:00:48.281319 19253 ts_manager.cc:194] Registered new tserver with Master: d55e029a4c054add86536445ef5d1b43 (127.1.166.3:35149)
I20260620 23:00:48.281689 19253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.3:49989
I20260620 23:00:48.289778 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:48.294945 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.291322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.1.166.62:41149)
0620 23:00:48.291393 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.291395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.291490 (+ 95us) server_negotiation.cc:407] Connection header received
0620 23:00:48.291555 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.291556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.291565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.291606 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:48.291762 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.292253 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.293393 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.293617 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.293683 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.294655 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:48.294657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:48.294657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:48.294663 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:48.294681 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:48.294685 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:48.294752 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.294836 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.294867 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:00:48.297258 19253 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:00:48.299145 19253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:48.306159 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.302037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.1.166.1:46747)
0620 23:00:48.302191 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.302195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.302204 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:00:48.302239 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.302241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.302255 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.302371 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:48.302483 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.303652 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.304910 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.305648 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.305726 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.305835 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.305995 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.306038 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260620 23:00:48.306120 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.301604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:35149 (local address 127.0.0.1:60210)
0620 23:00:48.301733 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.301774 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:48.301847 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:48.302377 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:48.302377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:48.302379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:48.302421 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.302422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.303449 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.303450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.304783 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.304787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.305772 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.305773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.305880 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.305895 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:00:48.305941 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 23:00:48.306003 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260620 23:00:48.306136 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.301824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46747 (local address 127.0.0.1:51228)
0620 23:00:48.302042 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.302051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:48.302102 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:48.302333 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:48.302334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:48.302356 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:48.302411 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.302414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.303744 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.303746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.304785 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.304787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.305745 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.305746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.305876 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.305894 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:00:48.305959 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:00:48.306011 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":23,"thread_start_us":67,"threads_started":1}
I20260620 23:00:48.306675 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.301787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.1.166.3:35149)
0620 23:00:48.302207 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.302210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.302219 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:00:48.302253 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.302255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.302268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.302486 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:48.302536 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.303361 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.304903 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.305671 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.306413 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.306510 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.306535 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.306589 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":91,"threads_started":1}
I20260620 23:00:48.307356 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.301605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45135 (local address 127.0.0.1:37932)
0620 23:00:48.301887 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.301897 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:48.301949 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:48.302223 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:48.302227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:48.302229 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:48.302312 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.302315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.304895 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.304896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.305931 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.305933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.307105 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.307107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.307171 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.307176 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:48.307232 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 23:00:48.307278 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":70,"threads_started":1}
I20260620 23:00:48.307404 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.301701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.1.166.2:45135)
0620 23:00:48.302056 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.302062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.302072 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:00:48.302116 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.302118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.302135 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.302209 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:48.302443 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.303652 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.306025 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.306906 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.306976 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.307065 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.307268 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.307318 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":103,"threads_started":1}
I20260620 23:00:48.307384 19375 tablet_service.cc:1467] Processing CreateTablet for tablet ef08b22f94d3423c9c8994ae641c61a1 (DEFAULT_TABLE table=test-table [id=be4ed62bb7404d4e8f20429d9d222369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:48.307439 19637 tablet_service.cc:1467] Processing CreateTablet for tablet ef08b22f94d3423c9c8994ae641c61a1 (DEFAULT_TABLE table=test-table [id=be4ed62bb7404d4e8f20429d9d222369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:48.307794 19637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef08b22f94d3423c9c8994ae641c61a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:48.307796 19375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef08b22f94d3423c9c8994ae641c61a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:48.308074 19506 tablet_service.cc:1467] Processing CreateTablet for tablet ef08b22f94d3423c9c8994ae641c61a1 (DEFAULT_TABLE table=test-table [id=be4ed62bb7404d4e8f20429d9d222369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:48.308398 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef08b22f94d3423c9c8994ae641c61a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:48.310760 19717 tablet_bootstrap.cc:492] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc: Bootstrap starting.
I20260620 23:00:48.311560 19717 tablet_bootstrap.cc:654] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:48.311843 19717 log.cc:824] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:48.312109 19719 tablet_bootstrap.cc:492] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: Bootstrap starting.
I20260620 23:00:48.312161 19718 tablet_bootstrap.cc:492] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: Bootstrap starting.
I20260620 23:00:48.312557 19717 tablet_bootstrap.cc:492] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc: No bootstrap required, opened a new log
I20260620 23:00:48.312624 19717 ts_tablet_manager.cc:1397] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:00:48.312803 19718 tablet_bootstrap.cc:654] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:48.312914 19719 tablet_bootstrap.cc:654] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:48.313191 19718 log.cc:824] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:48.313225 19719 log.cc:824] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:48.314004 19718 tablet_bootstrap.cc:492] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: No bootstrap required, opened a new log
I20260620 23:00:48.314086 19718 ts_tablet_manager.cc:1397] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:00:48.314201 19719 tablet_bootstrap.cc:492] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: No bootstrap required, opened a new log
I20260620 23:00:48.314270 19719 ts_tablet_manager.cc:1397] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:00:48.315124 19717 raft_consensus.cc:348] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.315287 19717 raft_consensus.cc:374] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:48.315320 19717 raft_consensus.cc:729] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1a3d6119b84a1eb3f58e912a4e9cfc, State: Initialized, Role: FOLLOWER
I20260620 23:00:48.315474 19717 consensus_queue.cc:260] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.315758 19717 ts_tablet_manager.cc:1428] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:00:48.315901 19441 heartbeater.cc:503] Master 127.1.166.62:41149 was elected leader, sending a full tablet report...
I20260620 23:00:48.317116 19719 raft_consensus.cc:348] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.317052 19718 raft_consensus.cc:348] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.317306 19718 raft_consensus.cc:374] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:48.317306 19719 raft_consensus.cc:374] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:48.317344 19719 raft_consensus.cc:729] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58681e133604e2cbd7d13dadd5080e4, State: Initialized, Role: FOLLOWER
I20260620 23:00:48.317344 19718 raft_consensus.cc:729] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55e029a4c054add86536445ef5d1b43, State: Initialized, Role: FOLLOWER
I20260620 23:00:48.317476 19718 consensus_queue.cc:260] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.317476 19719 consensus_queue.cc:260] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.317800 19718 ts_tablet_manager.cc:1428] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 23:00:48.317800 19719 ts_tablet_manager.cc:1428] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 23:00:48.317840 19703 heartbeater.cc:503] Master 127.1.166.62:41149 was elected leader, sending a full tablet report...
I20260620 23:00:48.317946 19572 heartbeater.cc:503] Master 127.1.166.62:41149 was elected leader, sending a full tablet report...
I20260620 23:00:48.330325 19724 raft_consensus.cc:482] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:48.330610 19724 raft_consensus.cc:504] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.331027 19724 leader_election.cc:288] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b58681e133604e2cbd7d13dadd5080e4 (127.1.166.2:45135), 8a1a3d6119b84a1eb3f58e912a4e9cfc (127.1.166.1:46747)
I20260620 23:00:48.333782 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.331257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:52949 (local address 127.1.166.2:45135)
0620 23:00:48.331305 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.331306 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.331335 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:00:48.331366 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.331366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.331380 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.331433 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:48.331628 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.332137 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.333086 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.333443 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.333520 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.333531 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.333659 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.333701 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:00:48.333830 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.331387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:59645 (local address 127.1.166.1:46747)
0620 23:00:48.331430 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.331432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.331479 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:00:48.331540 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.331541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.331554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.331606 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:48.331750 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.332245 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.333150 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.333516 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.333575 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.333584 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.333725 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.333762 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:00:48.333845 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.331186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45135 (local address 127.1.166.3:52949)
0620 23:00:48.331234 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.331246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:48.331288 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:48.331475 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:48.331476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:48.331479 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:48.331550 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.331553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.332219 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.332221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.332997 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.332999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.333521 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.333522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.333594 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.333597 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:48.333649 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 23:00:48.333708 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":13}
I20260620 23:00:48.334043 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.331293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46747 (local address 127.1.166.3:59645)
0620 23:00:48.331425 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.331434 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:48.331481 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:48.331610 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:48.331611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:48.331612 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:48.331681 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.331683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.332318 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.332318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.333073 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:48.333075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.333585 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.333586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:48.333668 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.333671 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:48.333721 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:00:48.333758 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260620 23:00:48.334252 19526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef08b22f94d3423c9c8994ae641c61a1" candidate_uuid: "d55e029a4c054add86536445ef5d1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58681e133604e2cbd7d13dadd5080e4" is_pre_election: true
I20260620 23:00:48.334436 19526 raft_consensus.cc:2454] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55e029a4c054add86536445ef5d1b43 in term 0.
I20260620 23:00:48.334388 19395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef08b22f94d3423c9c8994ae641c61a1" candidate_uuid: "d55e029a4c054add86536445ef5d1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" is_pre_election: true
I20260620 23:00:48.334547 19395 raft_consensus.cc:2454] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55e029a4c054add86536445ef5d1b43 in term 0.
I20260620 23:00:48.334653 19591 leader_election.cc:302] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b58681e133604e2cbd7d13dadd5080e4, d55e029a4c054add86536445ef5d1b43; no voters:
I20260620 23:00:48.334800 19724 raft_consensus.cc:2781] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:48.334882 19724 raft_consensus.cc:482] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:48.334903 19724 raft_consensus.cc:3037] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:48.335675 19724 raft_consensus.cc:504] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.335958 19724 leader_election.cc:288] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [CANDIDATE]: Term 1 election: Requested vote from peers b58681e133604e2cbd7d13dadd5080e4 (127.1.166.2:45135), 8a1a3d6119b84a1eb3f58e912a4e9cfc (127.1.166.1:46747)
I20260620 23:00:48.336081 19526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef08b22f94d3423c9c8994ae641c61a1" candidate_uuid: "d55e029a4c054add86536445ef5d1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58681e133604e2cbd7d13dadd5080e4"
I20260620 23:00:48.336172 19395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef08b22f94d3423c9c8994ae641c61a1" candidate_uuid: "d55e029a4c054add86536445ef5d1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc"
I20260620 23:00:48.336201 19526 raft_consensus.cc:3037] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:48.336261 19395 raft_consensus.cc:3037] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:48.337122 19395 raft_consensus.cc:2454] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55e029a4c054add86536445ef5d1b43 in term 1.
I20260620 23:00:48.337122 19526 raft_consensus.cc:2454] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55e029a4c054add86536445ef5d1b43 in term 1.
I20260620 23:00:48.337376 19591 leader_election.cc:302] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1a3d6119b84a1eb3f58e912a4e9cfc, d55e029a4c054add86536445ef5d1b43; no voters:
I20260620 23:00:48.337476 19724 raft_consensus.cc:2781] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:48.337642 19724 raft_consensus.cc:686] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 1 LEADER]: Becoming Leader. State: Replica: d55e029a4c054add86536445ef5d1b43, State: Running, Role: LEADER
I20260620 23:00:48.337731 19724 consensus_queue.cc:237] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } }
I20260620 23:00:48.338652 19253 catalog_manager.cc:5495] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to d55e029a4c054add86536445ef5d1b43 (127.1.166.3). New cstate: current_term: 1 leader_uuid: "d55e029a4c054add86536445ef5d1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55e029a4c054add86536445ef5d1b43" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 35149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:48.353535 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.351235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.1.166.3:35149)
0620 23:00:48.351297 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:48.351299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:48.351308 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:00:48.351348 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:48.351348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:48.351367 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:48.351433 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:00:48.351563 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.352050 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.352799 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:48.352998 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:48.353056 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:48.353191 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:00:48.353305 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:00:48.353380 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:48.353401 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 23:00:48.353423 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:00:48.355958 19526 raft_consensus.cc:1261] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 1 FOLLOWER]: Refusing update from remote peer d55e029a4c054add86536445ef5d1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:48.355958 19395 raft_consensus.cc:1261] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 1 FOLLOWER]: Refusing update from remote peer d55e029a4c054add86536445ef5d1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:48.356496 19724 consensus_queue.cc:1035] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58681e133604e2cbd7d13dadd5080e4" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:48.356740 19727 consensus_queue.cc:1035] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1a3d6119b84a1eb3f58e912a4e9cfc" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:48.360112 19732 mvcc.cc:205] Tried to move back new op lower bound from 7299057452461703168 to 7299057452392099840. Current Snapshot: MvccSnapshot[applied={T|T < 7299057452461703168}]
I20260620 23:00:48.363536 19253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:48.363663 19253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:48.364609 19253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a00e9f80e774aa5b282641f1ceed3b2: Sending DeleteTablet for 3 replicas of tablet ef08b22f94d3423c9c8994ae641c61a1
I20260620 23:00:48.365115 19375 tablet_service.cc:1514] Processing DeleteTablet for tablet ef08b22f94d3423c9c8994ae641c61a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:48 UTC) from {username='slave'} at 127.0.0.1:51228
I20260620 23:00:48.365160 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet ef08b22f94d3423c9c8994ae641c61a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:48 UTC) from {username='slave'} at 127.0.0.1:37932
I20260620 23:00:48.365577 19637 tablet_service.cc:1514] Processing DeleteTablet for tablet ef08b22f94d3423c9c8994ae641c61a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:48 UTC) from {username='slave'} at 127.0.0.1:60210
I20260620 23:00:48.365796 19740 tablet_replica.cc:330] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc: stopping tablet replica
I20260620 23:00:48.365928 19740 raft_consensus.cc:2229] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:48.365932 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 19313
I20260620 23:00:48.366062 19740 raft_consensus.cc:2258] T ef08b22f94d3423c9c8994ae641c61a1 P 8a1a3d6119b84a1eb3f58e912a4e9cfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:48.366250 19741 tablet_replica.cc:330] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: stopping tablet replica
I20260620 23:00:48.366396 19741 raft_consensus.cc:2229] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:48.366493 19741 raft_consensus.cc:2258] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:48.367637 19741 ts_tablet_manager.cc:1905] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:48.369069 19741 ts_tablet_manager.cc:1918] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:48.369153 19741 log.cc:1196] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/ef08b22f94d3423c9c8994ae641c61a1
I20260620 23:00:48.369379 19741 ts_tablet_manager.cc:1939] T ef08b22f94d3423c9c8994ae641c61a1 P b58681e133604e2cbd7d13dadd5080e4: Deleting consensus metadata
I20260620 23:00:48.369809 19238 catalog_manager.cc:4841] TS b58681e133604e2cbd7d13dadd5080e4 (127.1.166.2:45135): tablet ef08b22f94d3423c9c8994ae641c61a1 (table test-table [id=be4ed62bb7404d4e8f20429d9d222369]) successfully deleted
I20260620 23:00:48.370440 19742 tablet_replica.cc:330] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: stopping tablet replica
I20260620 23:00:48.370594 19742 raft_consensus.cc:2229] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:00:48.370750 19742 raft_consensus.cc:2258] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:48.371194 19742 ts_tablet_manager.cc:1905] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 23:00:48.372301 19238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:46747 (error 108)
I20260620 23:00:48.372467 19742 ts_tablet_manager.cc:1918] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:48.372538 19742 log.cc:1196] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/ef08b22f94d3423c9c8994ae641c61a1
I20260620 23:00:48.372553 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 19444
I20260620 23:00:48.372767 19742 ts_tablet_manager.cc:1939] T ef08b22f94d3423c9c8994ae641c61a1 P d55e029a4c054add86536445ef5d1b43: Deleting consensus metadata
I20260620 23:00:48.373661 19240 catalog_manager.cc:4841] TS d55e029a4c054add86536445ef5d1b43 (127.1.166.3:35149): tablet ef08b22f94d3423c9c8994ae641c61a1 (table test-table [id=be4ed62bb7404d4e8f20429d9d222369]) successfully deleted
I20260620 23:00:48.375018 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:48.374845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46747 (local address 127.0.0.1:51234)
0620 23:00:48.374910 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:48.374970 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:46747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:00:48.375244 19238 catalog_manager.cc:4584] TS 8a1a3d6119b84a1eb3f58e912a4e9cfc (127.1.166.1:46747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef08b22f94d3423c9c8994ae641c61a1: Network error: Client connection negotiation failed: client connection to 127.1.166.1:46747: connect: Connection refused (error 111)
I20260620 23:00:48.381018 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 19575
I20260620 23:00:48.386981 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 19222
2026-06-20T23:00:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (733 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:00:48 dist-test-slave-rdl9 krb5kdc[19746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:00:48 dist-test-slave-rdl9 krb5kdc[19746](info): set up 2 sockets
Jun 20 23:00:48 dist-test-slave-rdl9 krb5kdc[19746](info): commencing operation
krb5kdc: starting...
W20260620 23:00:49.390810 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:00:49 dist-test-slave-rdl9 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:00:49.515833 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:41447
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:41447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:49.608142 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:49.608511 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:49.608567 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:49.609906 19759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:49.609962 19759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:49.609982 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:49.609997 19759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:49.610013 19759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:49.611869 19759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:41447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:41447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:49.612174 19759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:49.612393 19759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:49.618036 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:49.618184 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:49.618184 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:49.618453 19759 server_base.cc:1034] running on GCE node
I20260620 23:00:49.618712 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:49.619263 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:49.620457 19759 hybrid_clock.cc:648] HybridClock initialized: now 1781996449620434 us; error 46 us; skew 500 ppm
Jun 20 23:00:49 dist-test-slave-rdl9 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996449, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:49.623934 19759 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:00:49.624495 19759 webserver.cc:460] Webserver started at http://127.1.166.62:46447/ using document root <none> and password file <none>
I20260620 23:00:49.624713 19759 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:49.624766 19759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:49.624897 19759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:49.626035 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "92fcaa009a8b479b9cb9087d94e7aa1a"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "bc74034a027170fab6214aa0e01a773a"
server_key_iv: "07095954b81262be3b793103d12ac113"
server_key_version: "encryptionkey@0"
I20260620 23:00:49.626582 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "92fcaa009a8b479b9cb9087d94e7aa1a"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "bc74034a027170fab6214aa0e01a773a"
server_key_iv: "07095954b81262be3b793103d12ac113"
server_key_version: "encryptionkey@0"
I20260620 23:00:49.628317 19759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:00:49.629182 19774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:49.629383 19759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:49.629482 19759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "92fcaa009a8b479b9cb9087d94e7aa1a"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "bc74034a027170fab6214aa0e01a773a"
server_key_iv: "07095954b81262be3b793103d12ac113"
server_key_version: "encryptionkey@0"
I20260620 23:00:49.629558 19759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:49.648644 19759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:49.651472 19759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:49.651659 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:49.656503 19759 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:41447
I20260620 23:00:49.656661 19826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:41447 every 8 connection(s)
I20260620 23:00:49.656960 19759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:49.658118 19827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:49.661278 19827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a: Bootstrap starting.
I20260620 23:00:49.662014 19827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:49.662320 19827 log.cc:824] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:49.662709 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 19759
I20260620 23:00:49.662854 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:49.663057 1688 external_mini_cluster.cc:1428] Setting key 965e2960285b5ad09c0b608aca305d10
I20260620 23:00:49.663594 19827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a: No bootstrap required, opened a new log
I20260620 23:00:49.665980 19827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fcaa009a8b479b9cb9087d94e7aa1a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41447 } }
I20260620 23:00:49.666190 19827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:49.666229 19827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fcaa009a8b479b9cb9087d94e7aa1a, State: Initialized, Role: FOLLOWER
I20260620 23:00:49.666400 19827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fcaa009a8b479b9cb9087d94e7aa1a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41447 } }
I20260620 23:00:49.666525 19827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:49.666559 19827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:49.666597 19827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:49.667503 19827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fcaa009a8b479b9cb9087d94e7aa1a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41447 } }
I20260620 23:00:49.667634 19827 leader_election.cc:302] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92fcaa009a8b479b9cb9087d94e7aa1a; no voters:
I20260620 23:00:49.667833 19827 leader_election.cc:288] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:49.667889 19832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:49.668071 19832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [term 1 LEADER]: Becoming Leader. State: Replica: 92fcaa009a8b479b9cb9087d94e7aa1a, State: Running, Role: LEADER
I20260620 23:00:49.668149 19827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:49.668175 19832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fcaa009a8b479b9cb9087d94e7aa1a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41447 } }
I20260620 23:00:49.670109 19833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92fcaa009a8b479b9cb9087d94e7aa1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fcaa009a8b479b9cb9087d94e7aa1a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41447 } } }
I20260620 23:00:49.670404 19833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:49.670684 19834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92fcaa009a8b479b9cb9087d94e7aa1a. Latest consensus state: current_term: 1 leader_uuid: "92fcaa009a8b479b9cb9087d94e7aa1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fcaa009a8b479b9cb9087d94e7aa1a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41447 } } }
I20260620 23:00:49.670814 19834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a [sys.catalog]: This master's current role is: LEADER
W20260620 23:00:49.672312 19846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:00:49.672389 19846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:00:49.672436 19848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:49.672603 19848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:00:49 dist-test-slave-rdl9 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:49.677948 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:49.663664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.1.166.62:41447)
0620 23:00:49.663886 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:49.663897 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:49.663934 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:00:49.665148 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:49.665159 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:49.665244 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:49.665414 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:49.666071 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:49.667783 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:49.669059 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:49.669713 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:49.669806 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:49.673263 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:49.673268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:49.673269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:49.673284 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:49.675434 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:49.675954 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:49.675958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:49.675959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:49.676124 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:49.676364 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:49.676365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:49.676366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:49.676804 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:49.676993 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:49.677499 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 23:00:49.677677 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260620 23:00:49.678295 19848 catalog_manager.cc:1347] Generated new cluster ID: fb2e7b81641b4880bb0a4ff8592efdc4
I20260620 23:00:49.678407 19848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:49.689514 19848 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:49.690438 19848 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:49.696166 19848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a: Generated new TSK 0
I20260620 23:00:49.696434 19848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 23:00:49.723503 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41447
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:49.811439 19853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:49.811695 19853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:49.811724 19853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:49.813136 19853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:49.813216 19853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:49.813249 19853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 23:00:49.815131 19853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:49.815444 19853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:49.815708 19853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:49.821221 19859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:49.821221 19858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:49.821221 19861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:49.821439 19853 server_base.cc:1034] running on GCE node
I20260620 23:00:49.821738 19853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:49.822037 19853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:49.823267 19853 hybrid_clock.cc:648] HybridClock initialized: now 1781996449823227 us; error 75 us; skew 500 ppm
Jun 20 23:00:49 dist-test-slave-rdl9 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996449, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:49.826774 19853 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 23:00:49.827621 19853 webserver.cc:460] Webserver started at http://127.1.166.1:35153/ using document root <none> and password file <none>
I20260620 23:00:49.827948 19853 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:49.828011 19853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:49.828173 19853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:49.829430 19853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "d570b74b55334819bb73e2a62b71e598"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "fa470f6039182ccbc8e218fff8517269"
server_key_iv: "199ff018c9c3fbf3a0440f7d9a36f014"
server_key_version: "encryptionkey@0"
I20260620 23:00:49.829844 19853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "d570b74b55334819bb73e2a62b71e598"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "fa470f6039182ccbc8e218fff8517269"
server_key_iv: "199ff018c9c3fbf3a0440f7d9a36f014"
server_key_version: "encryptionkey@0"
I20260620 23:00:49.831869 19853 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 23:00:49.833071 19868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:49.833299 19853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:49.833412 19853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "d570b74b55334819bb73e2a62b71e598"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "fa470f6039182ccbc8e218fff8517269"
server_key_iv: "199ff018c9c3fbf3a0440f7d9a36f014"
server_key_version: "encryptionkey@0"
I20260620 23:00:49.833500 19853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:49.859841 19853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:49.862694 19853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:49.862852 19853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:49.863077 19853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:49.863375 19853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:49.863408 19853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:49.863437 19853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:49.863463 19853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:49.869205 19853 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:38845
I20260620 23:00:49.869275 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:38845 every 8 connection(s)
I20260620 23:00:49.869575 19853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 23:00:49.870687 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 19853
I20260620 23:00:49.870828 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 23:00:49.871105 1688 external_mini_cluster.cc:1428] Setting key d06d254a133206e1e2c832d5d27b5843
Jun 20 23:00:49 dist-test-slave-rdl9 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996449, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
I20260620 23:00:49.881206 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:49.870409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:45971 (local address 127.1.166.62:41447)
0620 23:00:49.870465 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:49.870467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:49.871045 (+ 578us) server_negotiation.cc:407] Connection header received
0620 23:00:49.871632 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:49.871634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:49.871655 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:49.871750 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:49.872827 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:49.873563 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:49.875145 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:49.875366 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:49.875437 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:49.878283 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:49.878285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:49.878285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:49.878294 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:49.879684 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:49.880156 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:49.880158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:49.880158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:49.880222 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:49.880496 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:49.880497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:49.880497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:49.880662 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:49.880726 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:49.881007 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:00:49.881106 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:00:49.881479 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:49.870523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41447 (local address 127.1.166.1:45971)
0620 23:00:49.870849 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:49.870908 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:49.871451 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:49.871828 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:49.871832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:49.872453 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:49.872691 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:49.872697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:49.873705 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:49.873706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:49.875020 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:49.875025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:49.875462 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:49.875463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:49.875610 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:49.876349 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:49.876360 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:49.878147 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:49.879790 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:49.879804 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:49.879807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:49.880079 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:49.880278 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:49.880279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:49.880279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:49.880402 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:49.880724 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:49.880732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:49.880887 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:00:49.880983 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:00:49.881171 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
I20260620 23:00:49.882038 19982 heartbeater.cc:347] Connected to a master server at 127.1.166.62:41447
I20260620 23:00:49.882158 19982 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:49.882385 19982 heartbeater.cc:511] Master 127.1.166.62:41447 requested a full tablet report, sending...
I20260620 23:00:49.882964 19791 ts_manager.cc:194] Registered new tserver with Master: d570b74b55334819bb73e2a62b71e598 (127.1.166.1:38845)
I20260620 23:00:49.883754 19791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:45971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 23:00:49.898420 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41447
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:49.983554 19987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:49.983798 19987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:49.983825 19987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:49.985095 19987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:49.985153 19987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:49.985183 19987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 23:00:49.986835 19987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:49.987125 19987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:49.987380 19987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:00:49.992933 19987 server_base.cc:1034] running on GCE node
W20260620 23:00:49.992918 19992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:49.992944 19993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:49.992909 19995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:49.993387 19987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:49.993765 19987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:49.994969 19987 hybrid_clock.cc:648] HybridClock initialized: now 1781996449994948 us; error 34 us; skew 500 ppm
Jun 20 23:00:49 dist-test-slave-rdl9 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996449, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:49.997898 19987 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 23:00:49.998472 19987 webserver.cc:460] Webserver started at http://127.1.166.2:45635/ using document root <none> and password file <none>
I20260620 23:00:49.998716 19987 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:49.998769 19987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:49.998903 19987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:50.000027 19987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "62f586c4bd574abeb7a838a90192a9ab"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "03c2aa1dcc107fb3c30b7fbcc06bab42"
server_key_iv: "3b28d0d598ab478db6812ac5f082980a"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.000381 19987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "62f586c4bd574abeb7a838a90192a9ab"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "03c2aa1dcc107fb3c30b7fbcc06bab42"
server_key_iv: "3b28d0d598ab478db6812ac5f082980a"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.001899 19987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:00:50.002962 20002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.003221 19987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:00:50.003327 19987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "62f586c4bd574abeb7a838a90192a9ab"
format_stamp: "Formatted at 2026-06-20 23:00:49 on dist-test-slave-rdl9"
server_key: "03c2aa1dcc107fb3c30b7fbcc06bab42"
server_key_iv: "3b28d0d598ab478db6812ac5f082980a"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.003412 19987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:50.019411 19987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:50.021652 19987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:50.021808 19987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:50.022034 19987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:50.022430 19987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:50.022469 19987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.022500 19987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:50.022524 19987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.028538 19987 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:39931
I20260620 23:00:50.028609 20115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:39931 every 8 connection(s)
I20260620 23:00:50.028920 19987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 23:00:50.034488 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 19987
I20260620 23:00:50.034621 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 23:00:50.034873 1688 external_mini_cluster.cc:1428] Setting key 29e88037e63a5599e9215596ea418168
Jun 20 23:00:50 dist-test-slave-rdl9 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996449, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:50.039850 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.029805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:58237 (local address 127.1.166.62:41447)
0620 23:00:50.029890 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.029893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.029908 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:00:50.030487 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.030489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.030502 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.030550 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:50.031532 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.032039 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.033125 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.033322 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.033412 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.036134 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:50.036135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:50.036135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:50.036143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:50.037858 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:50.038465 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:50.038469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:50.038470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:50.038606 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:50.039085 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:50.039087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:50.039087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:50.039239 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:50.039307 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.039635 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.039740 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:00:50.040060 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.029631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41447 (local address 127.1.166.2:58237)
0620 23:00:50.029793 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.029837 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.030389 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.030623 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.030628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.031223 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:50.031436 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.031441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.032134 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.032135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.033033 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.033037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.033424 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.033426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.033553 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.034075 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:50.034084 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:50.036016 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:50.037967 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:50.037979 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:50.037982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:50.038252 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:50.038804 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:50.038807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:50.038807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:50.038963 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:50.039313 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:50.039321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:50.039482 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.039582 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.039793 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260620 23:00:50.040563 20116 heartbeater.cc:347] Connected to a master server at 127.1.166.62:41447
I20260620 23:00:50.040694 20116 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:50.040923 20116 heartbeater.cc:511] Master 127.1.166.62:41447 requested a full tablet report, sending...
I20260620 23:00:50.041433 19791 ts_manager.cc:194] Registered new tserver with Master: 62f586c4bd574abeb7a838a90192a9ab (127.1.166.2:39931)
I20260620 23:00:50.041832 19791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:58237
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 23:00:50.066391 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41447
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:50.157612 20121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:50.157894 20121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:50.157935 20121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:50.160063 20121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:50.160187 20121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:50.160243 20121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 23:00:50.163285 20121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:50.163592 20121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:50.163836 20121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:50.169046 20127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.169050 20129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.169106 20126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:50.169610 20121 server_base.cc:1034] running on GCE node
I20260620 23:00:50.169862 20121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:50.170151 20121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:50.171325 20121 hybrid_clock.cc:648] HybridClock initialized: now 1781996450171306 us; error 47 us; skew 500 ppm
Jun 20 23:00:50 dist-test-slave-rdl9 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:50.173990 20121 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 23:00:50.174741 20121 webserver.cc:460] Webserver started at http://127.1.166.3:35849/ using document root <none> and password file <none>
I20260620 23:00:50.175036 20121 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:50.175087 20121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:50.175206 20121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:50.176481 20121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "28bc494fcfa94bc3b37923ae8e0cca86"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "012cd16fa745aaac55812166f2bbeb1a"
server_key_iv: "7b07a2183832cba1d684bf690231b1d8"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.176838 20121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "28bc494fcfa94bc3b37923ae8e0cca86"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "012cd16fa745aaac55812166f2bbeb1a"
server_key_iv: "7b07a2183832cba1d684bf690231b1d8"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.178478 20121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:00:50.179550 20136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.179831 20121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:50.179939 20121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "28bc494fcfa94bc3b37923ae8e0cca86"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "012cd16fa745aaac55812166f2bbeb1a"
server_key_iv: "7b07a2183832cba1d684bf690231b1d8"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.180037 20121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:50.193413 20121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:50.195691 20121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:50.195816 20121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:50.196070 20121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:50.196461 20121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:50.196502 20121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.196538 20121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:50.196550 20121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.202782 20121 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:41055
I20260620 23:00:50.202858 20249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:41055 every 8 connection(s)
I20260620 23:00:50.203189 20121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:00:50 dist-test-slave-rdl9 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996450, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:50.213363 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 20121
I20260620 23:00:50.213481 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 23:00:50.213690 1688 external_mini_cluster.cc:1428] Setting key 2b06fb458d6f80867fab0b4cd891c130
I20260620 23:00:50.214903 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.204091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:32803 (local address 127.1.166.62:41447)
0620 23:00:50.204178 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.204179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.204381 (+ 202us) server_negotiation.cc:407] Connection header received
0620 23:00:50.204988 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.204989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.205003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.205079 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:50.206304 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.207033 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.208320 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.208548 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.208631 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.211073 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:50.211075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:50.211075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:50.211084 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:50.212831 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:50.213594 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:50.213596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:50.213596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:50.213669 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:50.213988 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:50.213989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:50.213989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:50.214160 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:50.214221 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.214665 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.214813 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:00:50.215327 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.204040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41447 (local address 127.1.166.3:32803)
0620 23:00:50.204226 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.204273 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.204862 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.205147 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.205151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.205872 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:50.206138 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.206145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.207171 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.207173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.208199 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.208202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.208685 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.208686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.208825 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.209389 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:50.209398 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:50.210969 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:50.213235 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:50.213244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:50.213246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:50.213512 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:50.213767 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:50.213768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:50.213768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:50.213927 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:50.214232 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:50.214240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:50.214488 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.214635 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.214851 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260620 23:00:50.215906 20250 heartbeater.cc:347] Connected to a master server at 127.1.166.62:41447
I20260620 23:00:50.216017 20250 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:50.216234 20250 heartbeater.cc:511] Master 127.1.166.62:41447 requested a full tablet report, sending...
I20260620 23:00:50.216652 19791 ts_manager.cc:194] Registered new tserver with Master: 28bc494fcfa94bc3b37923ae8e0cca86 (127.1.166.3:41055)
I20260620 23:00:50.217324 19791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:32803
I20260620 23:00:50.226176 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:50.234829 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.227786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.1.166.62:41447)
0620 23:00:50.227879 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.227881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.227920 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:00:50.227984 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.227985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.227995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.228044 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:50.228685 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.229211 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.229955 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.230162 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.230228 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.231680 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:50.231681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:50.231682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:50.231689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:50.233056 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:50.233555 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:50.233557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:50.233557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:50.233622 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:50.233965 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:50.233967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:50.233968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:50.234247 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:50.234432 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.234662 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.234752 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:00:50.237430 19791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:00:50.239246 19791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:50.246107 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.242115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.1.166.2:39931)
0620 23:00:50.242296 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.242300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.242311 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:00:50.242411 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.242413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.242429 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.242512 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:50.242802 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.243951 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.245183 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.245682 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.245746 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.245810 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.245955 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.246005 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260620 23:00:50.246130 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.242156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:38845 (local address 127.0.0.1:33420)
0620 23:00:50.242426 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.242433 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.242468 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.242664 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.242665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.242667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:50.242700 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.242702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.244080 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.244081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.244951 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.244953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.245897 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.245897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.245949 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.245952 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.245992 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.246036 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":56,"threads_started":1}
I20260620 23:00:50.246201 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.241934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.1.166.3:41055)
0620 23:00:50.242316 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.242321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.242333 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:00:50.242410 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.242412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.242428 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.242756 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:50.242825 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.243570 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.244994 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.245741 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.245983 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.246055 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.246080 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.246123 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":97,"threads_started":1}
I20260620 23:00:50.246138 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.241992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:39931 (local address 127.0.0.1:42152)
0620 23:00:50.242258 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.242267 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.242309 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.242544 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.242547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.242548 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:50.242725 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.242727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.244069 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.244070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.245109 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.245112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.245762 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.245763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.245857 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.245871 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.245933 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.246081 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":118,"thread_start_us":82,"threads_started":1}
I20260620 23:00:50.246474 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.241801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:41055 (local address 127.0.0.1:43816)
0620 23:00:50.241908 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.241948 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.242012 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.242554 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.242555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.242557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:50.242678 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.242682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.243669 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.243670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.244906 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.244909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.245839 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.245840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.245905 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.245909 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.245938 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.246011 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":23,"thread_start_us":49,"threads_started":1}
I20260620 23:00:50.246523 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.242280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.1.166.1:38845)
0620 23:00:50.242487 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.242491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.242502 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:00:50.242537 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.242538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.242551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.242623 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:50.242784 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.243972 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.245039 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.245741 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.245811 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.245869 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.246384 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.246433 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":104,"threads_started":1}
I20260620 23:00:50.247253 20184 tablet_service.cc:1467] Processing CreateTablet for tablet 9559807e8709478a9e6929a96a5cedf3 (DEFAULT_TABLE table=test-table [id=5871550ae6b54ba1aca3b393520cd495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:50.247275 19916 tablet_service.cc:1467] Processing CreateTablet for tablet 9559807e8709478a9e6929a96a5cedf3 (DEFAULT_TABLE table=test-table [id=5871550ae6b54ba1aca3b393520cd495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:50.247275 20050 tablet_service.cc:1467] Processing CreateTablet for tablet 9559807e8709478a9e6929a96a5cedf3 (DEFAULT_TABLE table=test-table [id=5871550ae6b54ba1aca3b393520cd495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:50.247619 20184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9559807e8709478a9e6929a96a5cedf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:50.247622 20050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9559807e8709478a9e6929a96a5cedf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:50.247622 19916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9559807e8709478a9e6929a96a5cedf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:50.250408 20267 tablet_bootstrap.cc:492] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: Bootstrap starting.
I20260620 23:00:50.251153 20267 tablet_bootstrap.cc:654] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:50.251170 20268 tablet_bootstrap.cc:492] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab: Bootstrap starting.
I20260620 23:00:50.251832 20268 tablet_bootstrap.cc:654] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:50.251845 20267 log.cc:824] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:50.252274 20269 tablet_bootstrap.cc:492] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598: Bootstrap starting.
I20260620 23:00:50.252411 20268 log.cc:824] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:50.252698 20267 tablet_bootstrap.cc:492] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: No bootstrap required, opened a new log
I20260620 23:00:50.252784 20267 ts_tablet_manager.cc:1397] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:50.252925 20269 tablet_bootstrap.cc:654] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:50.253288 20268 tablet_bootstrap.cc:492] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab: No bootstrap required, opened a new log
I20260620 23:00:50.253368 20268 ts_tablet_manager.cc:1397] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:50.253530 20269 log.cc:824] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:50.254475 20269 tablet_bootstrap.cc:492] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598: No bootstrap required, opened a new log
I20260620 23:00:50.254565 20269 ts_tablet_manager.cc:1397] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:50.254814 20267 raft_consensus.cc:348] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.255023 20267 raft_consensus.cc:374] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:50.255069 20267 raft_consensus.cc:729] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bc494fcfa94bc3b37923ae8e0cca86, State: Initialized, Role: FOLLOWER
I20260620 23:00:50.255198 20267 consensus_queue.cc:260] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.255509 20267 ts_tablet_manager.cc:1428] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 23:00:50.255540 20250 heartbeater.cc:503] Master 127.1.166.62:41447 was elected leader, sending a full tablet report...
I20260620 23:00:50.255505 20268 raft_consensus.cc:348] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.255697 20268 raft_consensus.cc:374] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:50.255738 20268 raft_consensus.cc:729] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f586c4bd574abeb7a838a90192a9ab, State: Initialized, Role: FOLLOWER
I20260620 23:00:50.255856 20268 consensus_queue.cc:260] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.256186 20116 heartbeater.cc:503] Master 127.1.166.62:41447 was elected leader, sending a full tablet report...
I20260620 23:00:50.256518 20268 ts_tablet_manager.cc:1428] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:50.257282 20269 raft_consensus.cc:348] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.257452 20269 raft_consensus.cc:374] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:50.257473 20269 raft_consensus.cc:729] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d570b74b55334819bb73e2a62b71e598, State: Initialized, Role: FOLLOWER
I20260620 23:00:50.257570 20269 consensus_queue.cc:260] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.257937 20269 ts_tablet_manager.cc:1428] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:50.257957 19982 heartbeater.cc:503] Master 127.1.166.62:41447 was elected leader, sending a full tablet report...
W20260620 23:00:50.279495 20117 tablet.cc:2369] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:00:50.317118 20275 raft_consensus.cc:482] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:50.317354 20275 raft_consensus.cc:504] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.318070 20275 leader_election.cc:288] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28bc494fcfa94bc3b37923ae8e0cca86 (127.1.166.3:41055), 62f586c4bd574abeb7a838a90192a9ab (127.1.166.2:39931)
I20260620 23:00:50.322000 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.318458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:41055 (local address 127.1.166.1:42705)
0620 23:00:50.318536 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.318552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.318637 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.318804 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.318805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.318807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:50.318892 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.318894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.319711 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.319712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.320627 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.320629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.321668 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.321669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.321763 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.321768 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.321838 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.321888 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:00:50.322079 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.318465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:42705 (local address 127.1.166.3:41055)
0620 23:00:50.318540 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.318542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.318619 (+ 77us) server_negotiation.cc:407] Connection header received
0620 23:00:50.318720 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.318721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.318736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.318781 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:50.318959 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.319631 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.320810 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.321552 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.321658 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.321678 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.321912 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.321978 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:00:50.322420 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.318867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:54165 (local address 127.1.166.2:39931)
0620 23:00:50.318920 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.318921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.319121 (+ 200us) server_negotiation.cc:407] Connection header received
0620 23:00:50.319179 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.319181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.319198 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.319277 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:50.319463 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.320290 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.321343 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.321836 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.321896 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.321926 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.322263 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.322317 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:00:50.322705 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.318735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:39931 (local address 127.1.166.1:54165)
0620 23:00:50.319022 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.319033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.319096 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.319301 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.319302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.319304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:50.319379 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.319381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.320406 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.320407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.321230 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.321233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.321959 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.321960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.322027 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.322032 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.322561 (+ 529us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.322627 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":217,"threads_started":1}
I20260620 23:00:50.322693 20204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9559807e8709478a9e6929a96a5cedf3" candidate_uuid: "d570b74b55334819bb73e2a62b71e598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" is_pre_election: true
I20260620 23:00:50.322933 20204 raft_consensus.cc:2454] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d570b74b55334819bb73e2a62b71e598 in term 0.
I20260620 23:00:50.323256 19869 leader_election.cc:302] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bc494fcfa94bc3b37923ae8e0cca86, d570b74b55334819bb73e2a62b71e598; no voters:
I20260620 23:00:50.323278 20070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9559807e8709478a9e6929a96a5cedf3" candidate_uuid: "d570b74b55334819bb73e2a62b71e598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f586c4bd574abeb7a838a90192a9ab" is_pre_election: true
I20260620 23:00:50.323450 20070 raft_consensus.cc:2454] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d570b74b55334819bb73e2a62b71e598 in term 0.
I20260620 23:00:50.323470 20275 raft_consensus.cc:2781] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:50.323534 20275 raft_consensus.cc:482] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:50.323567 20275 raft_consensus.cc:3037] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:50.324868 20275 raft_consensus.cc:504] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.325162 20275 leader_election.cc:288] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [CANDIDATE]: Term 1 election: Requested vote from peers 28bc494fcfa94bc3b37923ae8e0cca86 (127.1.166.3:41055), 62f586c4bd574abeb7a838a90192a9ab (127.1.166.2:39931)
I20260620 23:00:50.325290 20204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9559807e8709478a9e6929a96a5cedf3" candidate_uuid: "d570b74b55334819bb73e2a62b71e598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bc494fcfa94bc3b37923ae8e0cca86"
I20260620 23:00:50.325398 20204 raft_consensus.cc:3037] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:50.325376 20070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9559807e8709478a9e6929a96a5cedf3" candidate_uuid: "d570b74b55334819bb73e2a62b71e598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f586c4bd574abeb7a838a90192a9ab"
I20260620 23:00:50.325552 20070 raft_consensus.cc:3037] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:50.326603 20070 raft_consensus.cc:2454] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d570b74b55334819bb73e2a62b71e598 in term 1.
I20260620 23:00:50.326611 20204 raft_consensus.cc:2454] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d570b74b55334819bb73e2a62b71e598 in term 1.
I20260620 23:00:50.326823 19869 leader_election.cc:302] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f586c4bd574abeb7a838a90192a9ab, d570b74b55334819bb73e2a62b71e598; no voters:
I20260620 23:00:50.327026 20275 raft_consensus.cc:2781] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:50.327251 20275 raft_consensus.cc:686] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [term 1 LEADER]: Becoming Leader. State: Replica: d570b74b55334819bb73e2a62b71e598, State: Running, Role: LEADER
I20260620 23:00:50.327376 20275 consensus_queue.cc:237] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } }
I20260620 23:00:50.328567 19791 catalog_manager.cc:5495] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 reported cstate change: term changed from 0 to 1, leader changed from <none> to d570b74b55334819bb73e2a62b71e598 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "d570b74b55334819bb73e2a62b71e598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d570b74b55334819bb73e2a62b71e598" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 38845 } health_report { overall_health: HEALTHY } } }
I20260620 23:00:50.348381 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.345517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.1.166.1:38845)
0620 23:00:50.345595 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.345597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.345605 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:00:50.345644 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.345645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.345666 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.345728 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:00:50.345866 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.346380 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.347190 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.347446 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.347537 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.347839 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:00:50.347973 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:00:50.348079 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.348248 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.348262 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:00:50.351095 20070 raft_consensus.cc:1261] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab [term 1 FOLLOWER]: Refusing update from remote peer d570b74b55334819bb73e2a62b71e598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:50.351095 20204 raft_consensus.cc:1261] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 1 FOLLOWER]: Refusing update from remote peer d570b74b55334819bb73e2a62b71e598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:50.351603 20277 consensus_queue.cc:1035] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28bc494fcfa94bc3b37923ae8e0cca86" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 41055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:50.351759 20275 consensus_queue.cc:1035] T 9559807e8709478a9e6929a96a5cedf3 P d570b74b55334819bb73e2a62b71e598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f586c4bd574abeb7a838a90192a9ab" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:50.354750 20280 mvcc.cc:205] Tried to move back new op lower bound from 7299057460633481216 to 7299057460541894656. Current Snapshot: MvccSnapshot[applied={T|T < 7299057460633481216}]
I20260620 23:00:50.354976 20281 mvcc.cc:205] Tried to move back new op lower bound from 7299057460633481216 to 7299057460541894656. Current Snapshot: MvccSnapshot[applied={T|T < 7299057460633481216}]
I20260620 23:00:50.356127 20282 mvcc.cc:205] Tried to move back new op lower bound from 7299057460633481216 to 7299057460541894656. Current Snapshot: MvccSnapshot[applied={T|T < 7299057460633481216}]
I20260620 23:00:50.358295 19790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:50.358481 19790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:50.359539 19790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92fcaa009a8b479b9cb9087d94e7aa1a: Sending DeleteTablet for 3 replicas of tablet 9559807e8709478a9e6929a96a5cedf3
I20260620 23:00:50.359972 20184 tablet_service.cc:1514] Processing DeleteTablet for tablet 9559807e8709478a9e6929a96a5cedf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:50 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:43816
I20260620 23:00:50.360059 20050 tablet_service.cc:1514] Processing DeleteTablet for tablet 9559807e8709478a9e6929a96a5cedf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:50 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:42152
I20260620 23:00:50.360277 20289 tablet_replica.cc:330] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: stopping tablet replica
I20260620 23:00:50.360302 19916 tablet_service.cc:1514] Processing DeleteTablet for tablet 9559807e8709478a9e6929a96a5cedf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:50 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:33420
I20260620 23:00:50.360347 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 19853
I20260620 23:00:50.360409 20289 raft_consensus.cc:2229] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:50.360517 20289 raft_consensus.cc:2258] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:50.361644 20289 ts_tablet_manager.cc:1905] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:50.363461 20289 ts_tablet_manager.cc:1918] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:50.363565 20289 log.cc:1196] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/9559807e8709478a9e6929a96a5cedf3
I20260620 23:00:50.363842 20289 ts_tablet_manager.cc:1939] T 9559807e8709478a9e6929a96a5cedf3 P 28bc494fcfa94bc3b37923ae8e0cca86: Deleting consensus metadata
I20260620 23:00:50.364424 19775 catalog_manager.cc:4841] TS 28bc494fcfa94bc3b37923ae8e0cca86 (127.1.166.3:41055): tablet 9559807e8709478a9e6929a96a5cedf3 (table test-table [id=5871550ae6b54ba1aca3b393520cd495]) successfully deleted
W20260620 23:00:50.366418 19775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:38845 (error 108)
I20260620 23:00:50.367044 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.366912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:38845 (local address 127.0.0.1:33444)
0620 23:00:50.366961 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.367004 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:38845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 23:00:50.367059 20290 tablet_replica.cc:330] T 9559807e8709478a9e6929a96a5cedf3 P 62f586c4bd574abeb7a838a90192a9ab: stopping tablet replica
I20260620 23:00:50.367075 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 19987
W20260620 23:00:50.367215 19775 catalog_manager.cc:4584] TS d570b74b55334819bb73e2a62b71e598 (127.1.166.1:38845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9559807e8709478a9e6929a96a5cedf3: Network error: Client connection negotiation failed: client connection to 127.1.166.1:38845: connect: Connection refused (error 111)
I20260620 23:00:50.372004 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 20121
I20260620 23:00:50.372049 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.371903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:39931 (local address 127.0.0.1:42168)
0620 23:00:50.371962 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.372005 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.2:39931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 23:00:50.377707 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 19759
2026-06-20T23:00:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1993 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T23:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:50Z Disabled control of system clock
I20260620 23:00:50.408406 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:39013
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:39013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:00:50.501150 20295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:50.501523 20295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:50.501605 20295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:50.503050 20295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:50.503111 20295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:50.503125 20295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:50.503137 20295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:50.503152 20295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:50.504714 20295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:39013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:39013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:50.504971 20295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:50.505244 20295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:00:50.510612 20295 server_base.cc:1034] running on GCE node
W20260620 23:00:50.510517 20300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.510531 20301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.510517 20303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:50.511291 20295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:50.511674 20295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:50.512864 20295 hybrid_clock.cc:648] HybridClock initialized: now 1781996450512822 us; error 62 us; skew 500 ppm
I20260620 23:00:50.514535 20295 webserver.cc:460] Webserver started at http://127.1.166.62:43799/ using document root <none> and password file <none>
I20260620 23:00:50.514951 20295 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:50.515035 20295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:50.515187 20295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:50.516692 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "252cb2829c15485b8c53302d4e0e76f2"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "0eaa83ddd5e1388d25d1a0569c0543e5"
server_key_iv: "049bca6d2f1b09444aeccb911d328552"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.517163 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "252cb2829c15485b8c53302d4e0e76f2"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "0eaa83ddd5e1388d25d1a0569c0543e5"
server_key_iv: "049bca6d2f1b09444aeccb911d328552"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.519248 20295 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 23:00:50.520220 20309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.520447 20295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:50.520542 20295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "252cb2829c15485b8c53302d4e0e76f2"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "0eaa83ddd5e1388d25d1a0569c0543e5"
server_key_iv: "049bca6d2f1b09444aeccb911d328552"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.520622 20295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:50.529230 20295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:50.529511 20295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:50.529629 20295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:50.533290 20295 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:39013
I20260620 23:00:50.533324 20361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:39013 every 8 connection(s)
I20260620 23:00:50.533636 20295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:50.534255 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 20295
I20260620 23:00:50.534405 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:50.534629 1688 external_mini_cluster.cc:1428] Setting key 2480a9f7ffcb12a70ffb8a7cb62f69cf
I20260620 23:00:50.535022 20362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:50.537997 20362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2: Bootstrap starting.
I20260620 23:00:50.538614 20362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:50.538859 20362 log.cc:824] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:50.539474 20362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2: No bootstrap required, opened a new log
I20260620 23:00:50.540582 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.535267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.1.166.62:39013)
0620 23:00:50.535568 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.535578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.535611 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:00:50.536141 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.536155 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.536219 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.536343 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:50.536555 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.537683 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.538591 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.539170 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.539296 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.539971 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:50.539976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:50.539977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:50.540147 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:50.540183 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:50.540193 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:50.540247 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.540322 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.540402 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260620 23:00:50.541839 20362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252cb2829c15485b8c53302d4e0e76f2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39013 } }
I20260620 23:00:50.542009 20362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:50.542042 20362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252cb2829c15485b8c53302d4e0e76f2, State: Initialized, Role: FOLLOWER
I20260620 23:00:50.542169 20362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252cb2829c15485b8c53302d4e0e76f2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39013 } }
I20260620 23:00:50.542236 20362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:50.542268 20362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:50.542317 20362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:50.542991 20362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252cb2829c15485b8c53302d4e0e76f2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39013 } }
I20260620 23:00:50.543098 20362 leader_election.cc:302] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252cb2829c15485b8c53302d4e0e76f2; no voters:
I20260620 23:00:50.543282 20362 leader_election.cc:288] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:50.543339 20367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:50.543509 20367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [term 1 LEADER]: Becoming Leader. State: Replica: 252cb2829c15485b8c53302d4e0e76f2, State: Running, Role: LEADER
I20260620 23:00:50.543586 20362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:50.543617 20367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252cb2829c15485b8c53302d4e0e76f2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39013 } }
I20260620 23:00:50.543998 20369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252cb2829c15485b8c53302d4e0e76f2. Latest consensus state: current_term: 1 leader_uuid: "252cb2829c15485b8c53302d4e0e76f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252cb2829c15485b8c53302d4e0e76f2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39013 } } }
I20260620 23:00:50.544132 20369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:50.543993 20368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252cb2829c15485b8c53302d4e0e76f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252cb2829c15485b8c53302d4e0e76f2" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39013 } } }
I20260620 23:00:50.544291 20368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:50.544402 20376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:50.544931 20376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:50.546888 20376 catalog_manager.cc:1347] Generated new cluster ID: 5f3940952670419f8dd91a7097f8c733
I20260620 23:00:50.546947 20376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:50.564654 20376 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:50.565208 20376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:50.575284 20376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2: Generated new TSK 0
I20260620 23:00:50.575495 20376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:50.578107 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39013
--builtin_ntp_servers=127.1.166.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:00:50.672511 20386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:50.672799 20386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:50.672832 20386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:50.674610 20386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:50.674708 20386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:50.674747 20386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 23:00:50.676607 20386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:50.676951 20386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:50.677361 20386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:50.682768 20391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.683023 20394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.683085 20392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:50.683876 20386 server_base.cc:1034] running on GCE node
I20260620 23:00:50.684144 20386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:50.684466 20386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:50.685714 20386 hybrid_clock.cc:648] HybridClock initialized: now 1781996450685674 us; error 56 us; skew 500 ppm
I20260620 23:00:50.690548 20386 webserver.cc:460] Webserver started at http://127.1.166.1:43511/ using document root <none> and password file <none>
I20260620 23:00:50.690956 20386 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:50.691046 20386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:50.691161 20386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:50.692600 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "c6e92e86b61c41e9876190609f122f27"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "560130ab9977fa4adb832c135260bf76"
server_key_iv: "4b27b0bfb89136ef12b9a877d73e917b"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.693244 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "c6e92e86b61c41e9876190609f122f27"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "560130ab9977fa4adb832c135260bf76"
server_key_iv: "4b27b0bfb89136ef12b9a877d73e917b"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.694936 20386 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 23:00:50.696053 20400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.696338 20386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:50.696461 20386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "c6e92e86b61c41e9876190609f122f27"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "560130ab9977fa4adb832c135260bf76"
server_key_iv: "4b27b0bfb89136ef12b9a877d73e917b"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.696568 20386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:50.708364 20386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:50.708865 20386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:50.709028 20386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:50.709322 20386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:50.709720 20386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:50.709766 20386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.709801 20386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:50.709821 20386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.717398 20386 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:35993
I20260620 23:00:50.717471 20513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:35993 every 8 connection(s)
I20260620 23:00:50.717875 20386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 23:00:50.723703 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.718879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:55915 (local address 127.1.166.62:39013)
0620 23:00:50.718988 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.718990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.719253 (+ 263us) server_negotiation.cc:407] Connection header received
0620 23:00:50.719784 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.719786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.719804 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.719892 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:50.720581 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.721097 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.722172 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.722415 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.722473 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.723415 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:50.723416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:50.723416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:50.723422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:50.723438 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:50.723442 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:50.723487 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.723575 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.723612 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:00:50.723917 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.718832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39013 (local address 127.1.166.1:55915)
0620 23:00:50.719074 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.719125 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.719669 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.719935 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.719939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.719948 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:50.720483 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.720487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.721214 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.721216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.722074 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.722078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.722491 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.722492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.722640 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.723206 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:50.723206 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:50.723238 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:50.723291 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:50.723495 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:50.723501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:50.723504 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.723572 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.723685 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":117,"threads_started":1}
I20260620 23:00:50.724128 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 20386
I20260620 23:00:50.724303 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 23:00:50.724462 20514 heartbeater.cc:347] Connected to a master server at 127.1.166.62:39013
I20260620 23:00:50.724565 20514 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:50.724622 1688 external_mini_cluster.cc:1428] Setting key 7c2b1a81b35dd060f1a90639784a955c
I20260620 23:00:50.724781 20514 heartbeater.cc:511] Master 127.1.166.62:39013 requested a full tablet report, sending...
I20260620 23:00:50.725356 20325 ts_manager.cc:194] Registered new tserver with Master: c6e92e86b61c41e9876190609f122f27 (127.1.166.1:35993)
I20260620 23:00:50.725569 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39013
--builtin_ntp_servers=127.1.166.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:00:50.726207 20325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.1:55915
W20260620 23:00:50.816465 20517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:50.816824 20517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:50.816856 20517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:50.819205 20517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:50.819300 20517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:50.819339 20517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 23:00:50.821892 20517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:50.822237 20517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:50.822642 20517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:50.827728 20522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.828060 20525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.828240 20523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:50.828389 20517 server_base.cc:1034] running on GCE node
I20260620 23:00:50.828734 20517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:50.829108 20517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:50.830322 20517 hybrid_clock.cc:648] HybridClock initialized: now 1781996450830265 us; error 71 us; skew 500 ppm
I20260620 23:00:50.832216 20517 webserver.cc:460] Webserver started at http://127.1.166.2:33819/ using document root <none> and password file <none>
I20260620 23:00:50.832487 20517 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:50.832547 20517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:50.832684 20517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:50.833968 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "48cc2675333c4e74acf616b6373bfef9"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "4fe078b6f38de23c4ca7f21de8cb68d4"
server_key_iv: "c9ffd9e6ca40ea48b8a4fe999f7551c6"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.834470 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "48cc2675333c4e74acf616b6373bfef9"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "4fe078b6f38de23c4ca7f21de8cb68d4"
server_key_iv: "c9ffd9e6ca40ea48b8a4fe999f7551c6"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.836573 20517 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:50.837615 20531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.837909 20517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:50.838008 20517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "48cc2675333c4e74acf616b6373bfef9"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "4fe078b6f38de23c4ca7f21de8cb68d4"
server_key_iv: "c9ffd9e6ca40ea48b8a4fe999f7551c6"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.838114 20517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:50.854298 20517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:50.854965 20517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:50.855175 20517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:50.855440 20517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:50.855942 20517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:50.855997 20517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.856024 20517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:50.856034 20517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.861845 20517 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:42287
I20260620 23:00:50.861897 20644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:42287 every 8 connection(s)
I20260620 23:00:50.862250 20517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 23:00:50.867815 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.863002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:57841 (local address 127.1.166.62:39013)
0620 23:00:50.863088 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:50.863090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:50.863349 (+ 259us) server_negotiation.cc:407] Connection header received
0620 23:00:50.863841 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:50.863843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:50.863860 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:50.863952 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:50.864700 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.865230 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.866328 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.866601 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.866765 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.867569 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:50.867570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:50.867570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:50.867575 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:50.867590 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:50.867594 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:50.867644 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:50.867713 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 23:00:50.867746 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:00:50.868014 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:50.862984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39013 (local address 127.1.166.2:57841)
0620 23:00:50.863177 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:50.863213 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:50.863705 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:50.864021 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:50.864025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:50.864034 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:50.864557 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.864562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.865364 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.865365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.866205 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:50.866207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:50.866710 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:50.866712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:50.866846 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:50.867405 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:50.867405 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:50.867437 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:50.867465 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:50.867650 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:50.867653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:50.867655 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:00:50.867706 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:00:50.867804 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260620 23:00:50.868553 20645 heartbeater.cc:347] Connected to a master server at 127.1.166.62:39013
I20260620 23:00:50.868652 20645 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:50.868841 20645 heartbeater.cc:511] Master 127.1.166.62:39013 requested a full tablet report, sending...
I20260620 23:00:50.869211 20325 ts_manager.cc:194] Registered new tserver with Master: 48cc2675333c4e74acf616b6373bfef9 (127.1.166.2:42287)
I20260620 23:00:50.869593 20325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.2:57841
I20260620 23:00:50.871726 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 20517
I20260620 23:00:50.871819 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 23:00:50.871989 1688 external_mini_cluster.cc:1428] Setting key 65ca529cd9a7c816668dd837c2e142fe
I20260620 23:00:50.873301 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39013
--builtin_ntp_servers=127.1.166.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:00:50.963738 20648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:50.963989 20648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:50.964013 20648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:50.965301 20648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:50.965354 20648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:50.965384 20648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 23:00:50.966856 20648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:50.967101 20648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:50.967391 20648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:50.973166 20653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.973235 20656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:50.973250 20654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:50.973547 20648 server_base.cc:1034] running on GCE node
I20260620 23:00:50.973789 20648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:50.974074 20648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:50.975310 20648 hybrid_clock.cc:648] HybridClock initialized: now 1781996450975265 us; error 53 us; skew 500 ppm
I20260620 23:00:50.977247 20648 webserver.cc:460] Webserver started at http://127.1.166.3:37903/ using document root <none> and password file <none>
I20260620 23:00:50.977566 20648 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:50.977626 20648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:50.977771 20648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:50.979156 20648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "bcfbed4f3a0446cebf845fe7c502db57"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "0d33c2cbe0c0e627011be5bea8f09bbb"
server_key_iv: "1e6e96be334c16285e31fd4093ff9910"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.979550 20648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "bcfbed4f3a0446cebf845fe7c502db57"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "0d33c2cbe0c0e627011be5bea8f09bbb"
server_key_iv: "1e6e96be334c16285e31fd4093ff9910"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.981310 20648 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:50.982483 20662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:50.982683 20648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:50.982797 20648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "bcfbed4f3a0446cebf845fe7c502db57"
format_stamp: "Formatted at 2026-06-20 23:00:50 on dist-test-slave-rdl9"
server_key: "0d33c2cbe0c0e627011be5bea8f09bbb"
server_key_iv: "1e6e96be334c16285e31fd4093ff9910"
server_key_version: "encryptionkey@0"
I20260620 23:00:50.982890 20648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:51.004611 20648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:51.005180 20648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:51.005395 20648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:51.005649 20648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:51.006186 20648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:51.006234 20648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:51.006269 20648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:51.006287 20648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:51.012267 20648 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:40767
I20260620 23:00:51.012382 20775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:40767 every 8 connection(s)
I20260620 23:00:51.012753 20648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 23:00:51.018711 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.013738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:48757 (local address 127.1.166.62:39013)
0620 23:00:51.013852 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.013855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.014042 (+ 187us) server_negotiation.cc:407] Connection header received
0620 23:00:51.014676 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.014677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.014701 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.014759 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:51.015522 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.016037 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.017134 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.017338 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.017398 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.018413 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:51.018414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:51.018415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:51.018422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:51.018439 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:51.018444 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:51.018489 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.018571 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.018606 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:00:51.018956 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.013616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39013 (local address 127.1.166.3:48757)
0620 23:00:51.013911 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.013952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:51.014557 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:51.014840 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:51.014844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:51.014853 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:51.015420 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.015424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.016141 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.016142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.017026 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.017030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.017422 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.017423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.017574 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.018215 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:51.018216 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:51.018253 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:51.018283 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:51.018497 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:51.018503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:51.018507 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:00:51.018572 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:00:51.018702 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":115,"threads_started":1}
I20260620 23:00:51.019131 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 20648
I20260620 23:00:51.019232 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 23:00:51.019459 1688 external_mini_cluster.cc:1428] Setting key 2719e8e1caeacc0d2b31cf9482dab191
I20260620 23:00:51.019575 20776 heartbeater.cc:347] Connected to a master server at 127.1.166.62:39013
I20260620 23:00:51.019726 20776 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:51.019984 20776 heartbeater.cc:511] Master 127.1.166.62:39013 requested a full tablet report, sending...
I20260620 23:00:51.020435 20325 ts_manager.cc:194] Registered new tserver with Master: bcfbed4f3a0446cebf845fe7c502db57 (127.1.166.3:40767)
I20260620 23:00:51.020874 20325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.3:48757
I20260620 23:00:51.032294 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:51.037837 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.033938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.1.166.62:39013)
0620 23:00:51.034024 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.034025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.034040 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:00:51.034107 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.034108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.034118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.034170 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:51.034485 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.035073 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.035990 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.036220 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.036295 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.037293 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:51.037295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:51.037295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:51.037303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:51.037323 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:51.037328 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:51.037525 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.037711 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.037734 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:00:51.039886 20325 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:00:51.041649 20325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:51.048974 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.044637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.1.166.1:35993)
0620 23:00:51.044798 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.044803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.045079 (+ 276us) server_negotiation.cc:407] Connection header received
0620 23:00:51.045159 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.045161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.045177 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.045244 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:51.045429 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.046187 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.047615 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.048290 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.048372 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.048477 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.048799 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.048858 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260620 23:00:51.048983 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.044460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:35993 (local address 127.0.0.1:39666)
0620 23:00:51.045013 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.045022 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:51.045060 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:51.045303 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:51.045304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:51.045306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:51.045356 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.045357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.046277 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.046278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.047497 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.047501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.048446 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.048448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.048622 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.048650 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:00:51.048754 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:00:51.048851 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":30,"thread_start_us":33,"threads_started":1}
I20260620 23:00:51.049237 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.044244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42287 (local address 127.0.0.1:42550)
0620 23:00:51.044483 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.044539 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:51.044610 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:51.044803 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:51.044806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:51.044809 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:51.044911 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.044915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.046178 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.046178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.047682 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.047685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.048936 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.048937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.049014 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.049021 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:00:51.049074 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 23:00:51.049141 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":77,"thread_start_us":65,"threads_started":1}
I20260620 23:00:51.049508 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.044389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.1.166.2:42287)
0620 23:00:51.044643 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.044646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.044655 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:00:51.044689 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.044690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.044702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.044782 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:51.044992 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.046077 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.048034 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.048721 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.048796 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.048888 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.049361 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.049424 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":73,"threads_started":1}
I20260620 23:00:51.050714 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.044244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:40767 (local address 127.0.0.1:40664)
0620 23:00:51.044882 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.044893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:51.044950 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:51.045483 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:51.045484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:51.045487 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:51.045517 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.045519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.046743 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.046744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.048213 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.048217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.050428 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.050430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.050510 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.050516 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:00:51.050589 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:00:51.050639 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260620 23:00:51.050561 20448 tablet_service.cc:1467] Processing CreateTablet for tablet 0b752b0c552e49baba537fb94724be20 (DEFAULT_TABLE table=test-table [id=14cd77fd7f7d4c5ab93b5958398c5052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:51.050755 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.044319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.1.166.3:40767)
0620 23:00:51.044536 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.044541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.045110 (+ 569us) server_negotiation.cc:407] Connection header received
0620 23:00:51.045162 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.045164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.045349 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.045419 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:51.045598 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.046654 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.048493 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.050249 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.050403 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.050519 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.050608 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.050662 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":94,"threads_started":1}
I20260620 23:00:51.051066 20448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b752b0c552e49baba537fb94724be20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:51.051525 20710 tablet_service.cc:1467] Processing CreateTablet for tablet 0b752b0c552e49baba537fb94724be20 (DEFAULT_TABLE table=test-table [id=14cd77fd7f7d4c5ab93b5958398c5052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:51.051887 20710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b752b0c552e49baba537fb94724be20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:51.055359 20790 tablet_bootstrap.cc:492] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27: Bootstrap starting.
I20260620 23:00:51.055408 20579 tablet_service.cc:1467] Processing CreateTablet for tablet 0b752b0c552e49baba537fb94724be20 (DEFAULT_TABLE table=test-table [id=14cd77fd7f7d4c5ab93b5958398c5052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:51.055828 20579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b752b0c552e49baba537fb94724be20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:51.056150 20791 tablet_bootstrap.cc:492] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Bootstrap starting.
I20260620 23:00:51.056203 20790 tablet_bootstrap.cc:654] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:51.056519 20790 log.cc:824] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:51.056744 20791 tablet_bootstrap.cc:654] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:51.057003 20791 log.cc:824] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:51.057849 20791 tablet_bootstrap.cc:492] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: No bootstrap required, opened a new log
I20260620 23:00:51.057924 20791 ts_tablet_manager.cc:1397] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:00:51.059091 20794 tablet_bootstrap.cc:492] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Bootstrap starting.
I20260620 23:00:51.059136 20790 tablet_bootstrap.cc:492] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27: No bootstrap required, opened a new log
I20260620 23:00:51.059208 20790 ts_tablet_manager.cc:1397] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:00:51.059953 20794 tablet_bootstrap.cc:654] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:51.060299 20794 log.cc:824] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:51.060568 20791 raft_consensus.cc:348] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.060727 20791 raft_consensus.cc:374] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:51.060750 20791 raft_consensus.cc:729] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfbed4f3a0446cebf845fe7c502db57, State: Initialized, Role: FOLLOWER
I20260620 23:00:51.060905 20791 consensus_queue.cc:260] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.061169 20791 ts_tablet_manager.cc:1428] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 23:00:51.061273 20794 tablet_bootstrap.cc:492] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: No bootstrap required, opened a new log
I20260620 23:00:51.061313 20776 heartbeater.cc:503] Master 127.1.166.62:39013 was elected leader, sending a full tablet report...
I20260620 23:00:51.061420 20794 ts_tablet_manager.cc:1397] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:51.061601 20790 raft_consensus.cc:348] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.061787 20790 raft_consensus.cc:374] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:51.061833 20790 raft_consensus.cc:729] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e92e86b61c41e9876190609f122f27, State: Initialized, Role: FOLLOWER
I20260620 23:00:51.061944 20790 consensus_queue.cc:260] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.062227 20790 ts_tablet_manager.cc:1428] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:51.062424 20514 heartbeater.cc:503] Master 127.1.166.62:39013 was elected leader, sending a full tablet report...
I20260620 23:00:51.063841 20794 raft_consensus.cc:348] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.063993 20794 raft_consensus.cc:374] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:51.064015 20794 raft_consensus.cc:729] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc2675333c4e74acf616b6373bfef9, State: Initialized, Role: FOLLOWER
I20260620 23:00:51.064121 20794 consensus_queue.cc:260] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.064396 20794 ts_tablet_manager.cc:1428] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:51.064567 20645 heartbeater.cc:503] Master 127.1.166.62:39013 was elected leader, sending a full tablet report...
W20260620 23:00:51.112911 20646 tablet.cc:2369] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:00:51.218786 20515 tablet.cc:2369] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:00:51.263468 20777 tablet.cc:2369] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:00:51.541714 20797 raft_consensus.cc:482] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:51.541893 20797 raft_consensus.cc:504] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.542271 20797 leader_election.cc:288] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc2675333c4e74acf616b6373bfef9 (127.1.166.2:42287), bcfbed4f3a0446cebf845fe7c502db57 (127.1.166.3:40767)
I20260620 23:00:51.546000 20798 raft_consensus.cc:482] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:51.546171 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.542497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:40767 (local address 127.1.166.1:38405)
0620 23:00:51.542637 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.542648 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:51.542721 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:51.542846 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:51.542847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:51.542850 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:51.542956 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.542958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.543613 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.543613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.545005 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.545007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.545874 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.545875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.545969 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.545974 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:51.546024 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:00:51.546072 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":44,"thread_start_us":63,"threads_started":1}
I20260620 23:00:51.546201 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.542570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:38405 (local address 127.1.166.3:40767)
0620 23:00:51.542625 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.542627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.542720 (+ 93us) server_negotiation.cc:407] Connection header received
0620 23:00:51.542774 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.542775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.542785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.542839 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:51.543019 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.543534 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.545229 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.545690 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.545838 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.545852 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.546056 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.546139 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:00:51.546649 20727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b752b0c552e49baba537fb94724be20" candidate_uuid: "c6e92e86b61c41e9876190609f122f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfbed4f3a0446cebf845fe7c502db57" is_pre_election: true
I20260620 23:00:51.546828 20727 raft_consensus.cc:2454] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6e92e86b61c41e9876190609f122f27 in term 0.
I20260620 23:00:51.546685 20798 raft_consensus.cc:504] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.547130 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.542573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:60923 (local address 127.1.166.2:42287)
0620 23:00:51.542622 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.542624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.542773 (+ 149us) server_negotiation.cc:407] Connection header received
0620 23:00:51.542832 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.542833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.542844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.542895 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:51.543033 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.543635 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.545909 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.546485 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.546572 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.546591 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.546988 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.547066 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:00:51.547086 20403 leader_election.cc:302] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcfbed4f3a0446cebf845fe7c502db57, c6e92e86b61c41e9876190609f122f27; no voters:
I20260620 23:00:51.547196 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.542502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:42287 (local address 127.1.166.1:60923)
0620 23:00:51.542700 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.542710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:51.542771 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:51.542898 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:51.542899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:51.542900 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:51.542969 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.542971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.543722 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.543723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.544995 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.544998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.546645 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.546648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.546758 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.546764 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:00:51.546972 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 23:00:51.547051 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":26,"thread_start_us":61,"threads_started":1}
I20260620 23:00:51.547291 20797 raft_consensus.cc:2781] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:51.547351 20797 raft_consensus.cc:482] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:51.547375 20797 raft_consensus.cc:3037] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:51.547487 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b752b0c552e49baba537fb94724be20" candidate_uuid: "c6e92e86b61c41e9876190609f122f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc2675333c4e74acf616b6373bfef9" is_pre_election: true
I20260620 23:00:51.547605 20599 raft_consensus.cc:2454] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6e92e86b61c41e9876190609f122f27 in term 0.
I20260620 23:00:51.547665 20798 leader_election.cc:288] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6e92e86b61c41e9876190609f122f27 (127.1.166.1:35993), bcfbed4f3a0446cebf845fe7c502db57 (127.1.166.3:40767)
I20260620 23:00:51.548415 20797 raft_consensus.cc:504] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.548722 20797 leader_election.cc:288] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc2675333c4e74acf616b6373bfef9 (127.1.166.2:42287), bcfbed4f3a0446cebf845fe7c502db57 (127.1.166.3:40767)
I20260620 23:00:51.548996 20727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b752b0c552e49baba537fb94724be20" candidate_uuid: "c6e92e86b61c41e9876190609f122f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfbed4f3a0446cebf845fe7c502db57"
I20260620 23:00:51.549124 20727 raft_consensus.cc:3037] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:51.549115 20599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b752b0c552e49baba537fb94724be20" candidate_uuid: "c6e92e86b61c41e9876190609f122f27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc2675333c4e74acf616b6373bfef9"
I20260620 23:00:51.549201 20599 raft_consensus.cc:3037] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:51.550010 20727 raft_consensus.cc:2454] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6e92e86b61c41e9876190609f122f27 in term 1.
I20260620 23:00:51.550010 20599 raft_consensus.cc:2454] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6e92e86b61c41e9876190609f122f27 in term 1.
I20260620 23:00:51.550211 20403 leader_election.cc:302] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcfbed4f3a0446cebf845fe7c502db57, c6e92e86b61c41e9876190609f122f27; no voters:
I20260620 23:00:51.550324 20797 raft_consensus.cc:2781] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:51.550542 20797 raft_consensus.cc:686] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 1 LEADER]: Becoming Leader. State: Replica: c6e92e86b61c41e9876190609f122f27, State: Running, Role: LEADER
I20260620 23:00:51.550657 20797 consensus_queue.cc:237] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } }
I20260620 23:00:51.551151 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.547705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:56951 (local address 127.1.166.1:35993)
0620 23:00:51.547745 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.547747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.547823 (+ 76us) server_negotiation.cc:407] Connection header received
0620 23:00:51.547912 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.547913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.547921 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.547947 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:51.548138 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.548779 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.550249 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.550778 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.550845 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.550858 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.551039 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.551079 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:00:51.551165 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.547480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:35993 (local address 127.1.166.2:56951)
0620 23:00:51.547765 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.547776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:51.547848 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:51.548003 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:51.548004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:51.548007 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:51.548068 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.548070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.548855 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.548856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.550176 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.550179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.550894 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.550895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.550972 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.550978 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:00:51.551032 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:00:51.551079 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":26,"thread_start_us":71,"threads_started":1}
I20260620 23:00:51.551342 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.547910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:39109 (local address 127.1.166.3:40767)
0620 23:00:51.547949 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.547951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.547999 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:00:51.548057 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.548058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.548066 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.548109 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:51.548236 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.548995 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.550455 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.550943 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.550990 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.551001 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.551220 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.551261 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:00:51.551438 20468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b752b0c552e49baba537fb94724be20" candidate_uuid: "48cc2675333c4e74acf616b6373bfef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e92e86b61c41e9876190609f122f27" is_pre_election: true
I20260620 23:00:51.551600 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.547779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:40767 (local address 127.1.166.2:39109)
0620 23:00:51.547948 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.547957 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:51.548011 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:51.548110 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:51.548111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:51.548112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:51.548180 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.548181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.549091 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.549092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.550367 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:51.550370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.551071 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.551072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:51.551139 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.551143 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:00:51.551478 (+ 335us) client_negotiation.cc:240] Negotiation successful
0620 23:00:51.551527 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":16,"thread_start_us":42,"threads_started":1}
I20260620 23:00:51.551839 20727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b752b0c552e49baba537fb94724be20" candidate_uuid: "48cc2675333c4e74acf616b6373bfef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfbed4f3a0446cebf845fe7c502db57" is_pre_election: true
I20260620 23:00:51.551941 20727 raft_consensus.cc:2379] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48cc2675333c4e74acf616b6373bfef9 in current term 1: Already voted for candidate c6e92e86b61c41e9876190609f122f27 in this term.
I20260620 23:00:51.552182 20534 leader_election.cc:302] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48cc2675333c4e74acf616b6373bfef9; no voters: bcfbed4f3a0446cebf845fe7c502db57, c6e92e86b61c41e9876190609f122f27
I20260620 23:00:51.552325 20798 raft_consensus.cc:2726] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:00:51.552559 20326 catalog_manager.cc:5495] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6e92e86b61c41e9876190609f122f27 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "c6e92e86b61c41e9876190609f122f27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6e92e86b61c41e9876190609f122f27" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:51.605305 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.602442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.1.166.1:35993)
0620 23:00:51.602508 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:51.602510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:51.602521 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:00:51.602577 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:51.602578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:51.602603 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:51.602705 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:00:51.602913 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.603522 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.604309 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:51.604606 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:51.604679 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:51.604912 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:00:51.605022 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:00:51.605115 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:51.605224 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:00:51.605236 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:00:51.607095 20727 raft_consensus.cc:1261] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 1 FOLLOWER]: Refusing update from remote peer c6e92e86b61c41e9876190609f122f27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:51.607111 20599 raft_consensus.cc:1261] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 1 FOLLOWER]: Refusing update from remote peer c6e92e86b61c41e9876190609f122f27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:51.607379 20797 consensus_queue.cc:1035] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcfbed4f3a0446cebf845fe7c502db57" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:51.607533 20803 consensus_queue.cc:1035] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc2675333c4e74acf616b6373bfef9" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 42287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:51.609962 20812 mvcc.cc:205] Tried to move back new op lower bound from 7299057465779888128 to 7299057465552588800. Current Snapshot: MvccSnapshot[applied={T|T < 7299057465779888128}]
I20260620 23:00:51.610787 20813 mvcc.cc:205] Tried to move back new op lower bound from 7299057465779888128 to 7299057465552588800. Current Snapshot: MvccSnapshot[applied={T|T < 7299057465779888128}]
I20260620 23:00:51.612485 20326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:51.612588 20326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:51.613710 20326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 252cb2829c15485b8c53302d4e0e76f2: Sending DeleteTablet for 3 replicas of tablet 0b752b0c552e49baba537fb94724be20
I20260620 23:00:51.614096 20579 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b752b0c552e49baba537fb94724be20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:51 UTC) from {username='slave'} at 127.0.0.1:42550
I20260620 23:00:51.614218 20448 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b752b0c552e49baba537fb94724be20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:51 UTC) from {username='slave'} at 127.0.0.1:39666
I20260620 23:00:51.614477 20821 tablet_replica.cc:330] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27: stopping tablet replica
I20260620 23:00:51.614493 20710 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b752b0c552e49baba537fb94724be20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:51 UTC) from {username='slave'} at 127.0.0.1:40664
I20260620 23:00:51.614602 20821 raft_consensus.cc:2229] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:00:51.614737 20821 raft_consensus.cc:2258] T 0b752b0c552e49baba537fb94724be20 P c6e92e86b61c41e9876190609f122f27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:51.614845 20822 tablet_replica.cc:330] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: stopping tablet replica
I20260620 23:00:51.614956 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 20386
I20260620 23:00:51.615916 20820 tablet_replica.cc:330] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: stopping tablet replica
I20260620 23:00:51.616032 20820 raft_consensus.cc:2229] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:51.616147 20820 raft_consensus.cc:2258] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:51.616297 20822 raft_consensus.cc:2229] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:51.616406 20822 raft_consensus.cc:2258] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:51.616873 20822 ts_tablet_manager.cc:1905] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:51.618255 20822 ts_tablet_manager.cc:1918] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:51.618377 20822 log.cc:1196] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/0b752b0c552e49baba537fb94724be20
I20260620 23:00:51.618417 20820 ts_tablet_manager.cc:1905] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:51.618662 20822 ts_tablet_manager.cc:1939] T 0b752b0c552e49baba537fb94724be20 P bcfbed4f3a0446cebf845fe7c502db57: Deleting consensus metadata
I20260620 23:00:51.619236 20312 catalog_manager.cc:4841] TS bcfbed4f3a0446cebf845fe7c502db57 (127.1.166.3:40767): tablet 0b752b0c552e49baba537fb94724be20 (table test-table [id=14cd77fd7f7d4c5ab93b5958398c5052]) successfully deleted
I20260620 23:00:51.619889 20820 ts_tablet_manager.cc:1918] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:51.619956 20820 log.cc:1196] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/0b752b0c552e49baba537fb94724be20
I20260620 23:00:51.620132 20820 ts_tablet_manager.cc:1939] T 0b752b0c552e49baba537fb94724be20 P 48cc2675333c4e74acf616b6373bfef9: Deleting consensus metadata
I20260620 23:00:51.620613 20310 catalog_manager.cc:4841] TS 48cc2675333c4e74acf616b6373bfef9 (127.1.166.2:42287): tablet 0b752b0c552e49baba537fb94724be20 (table test-table [id=14cd77fd7f7d4c5ab93b5958398c5052]) successfully deleted
W20260620 23:00:51.621827 20310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:35993 (error 108)
I20260620 23:00:51.622053 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 20517
I20260620 23:00:51.626648 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:51.622220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:35993 (local address 127.0.0.1:39686)
0620 23:00:51.626505 (+ 4285us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:51.626597 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4241,"thread_start_us":66,"threads_started":1}
W20260620 23:00:51.626832 20310 catalog_manager.cc:4584] TS c6e92e86b61c41e9876190609f122f27 (127.1.166.1:35993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b752b0c552e49baba537fb94724be20: Network error: Client connection negotiation failed: client connection to 127.1.166.1:35993: connect: Connection refused (error 111)
I20260620 23:00:51.627259 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 20648
I20260620 23:00:51.632046 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 20295
2026-06-20T23:00:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1253 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:00:51 dist-test-slave-rdl9 krb5kdc[20827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:00:51 dist-test-slave-rdl9 krb5kdc[20827](info): set up 2 sockets
Jun 20 23:00:51 dist-test-slave-rdl9 krb5kdc[20827](info): commencing operation
krb5kdc: starting...
W20260620 23:00:52.606578 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:00:52 dist-test-slave-rdl9 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:00:52.731429 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:44505
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:44505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:52.820542 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:52.820806 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:52.820838 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:52.822175 20840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:52.822242 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:52.822261 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:52.822278 20840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:52.822293 20840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:52.824167 20840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:44505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:44505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:52.824465 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:52.824682 20840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:52.830168 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:52.830179 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:52.830185 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:52.830991 20840 server_base.cc:1034] running on GCE node
I20260620 23:00:52.831293 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:52.831643 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:52.832832 20840 hybrid_clock.cc:648] HybridClock initialized: now 1781996452832806 us; error 41 us; skew 500 ppm
Jun 20 23:00:52 dist-test-slave-rdl9 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:52.835728 20840 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:00:52.836290 20840 webserver.cc:460] Webserver started at http://127.1.166.62:35721/ using document root <none> and password file <none>
I20260620 23:00:52.836536 20840 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:52.836591 20840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:52.836725 20840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:52.837855 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "f356a78465ed48b68b9f721ae862cf59"
format_stamp: "Formatted at 2026-06-20 23:00:52 on dist-test-slave-rdl9"
server_key: "dcd5c5b73085f12bed8a7940c7993aca"
server_key_iv: "44e70415832c4b0a2075d0f628e0c677"
server_key_version: "encryptionkey@0"
I20260620 23:00:52.838233 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "f356a78465ed48b68b9f721ae862cf59"
format_stamp: "Formatted at 2026-06-20 23:00:52 on dist-test-slave-rdl9"
server_key: "dcd5c5b73085f12bed8a7940c7993aca"
server_key_iv: "44e70415832c4b0a2075d0f628e0c677"
server_key_version: "encryptionkey@0"
I20260620 23:00:52.839773 20840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:52.840724 20855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:52.840941 20840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:00:52.841041 20840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "f356a78465ed48b68b9f721ae862cf59"
format_stamp: "Formatted at 2026-06-20 23:00:52 on dist-test-slave-rdl9"
server_key: "dcd5c5b73085f12bed8a7940c7993aca"
server_key_iv: "44e70415832c4b0a2075d0f628e0c677"
server_key_version: "encryptionkey@0"
I20260620 23:00:52.841117 20840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:52.851616 20840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:52.853856 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:52.854020 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:52.858788 20840 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:44505
I20260620 23:00:52.858816 20907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:44505 every 8 connection(s)
I20260620 23:00:52.859326 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:52.860522 20908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:52.863114 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59: Bootstrap starting.
I20260620 23:00:52.863721 20908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:52.863969 20908 log.cc:824] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:52.864812 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59: No bootstrap required, opened a new log
I20260620 23:00:52.866627 20908 raft_consensus.cc:348] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356a78465ed48b68b9f721ae862cf59" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44505 } }
I20260620 23:00:52.866802 20908 raft_consensus.cc:374] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:52.866824 20908 raft_consensus.cc:729] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f356a78465ed48b68b9f721ae862cf59, State: Initialized, Role: FOLLOWER
I20260620 23:00:52.866957 20908 consensus_queue.cc:260] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356a78465ed48b68b9f721ae862cf59" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44505 } }
I20260620 23:00:52.867039 20908 raft_consensus.cc:388] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:52.867069 20908 raft_consensus.cc:482] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:52.867103 20908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:52.867337 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 20840
I20260620 23:00:52.867460 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:52.867633 1688 external_mini_cluster.cc:1428] Setting key f6ffef9d1aafdb01c7a0536aedb310e0
I20260620 23:00:52.867909 20908 raft_consensus.cc:504] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356a78465ed48b68b9f721ae862cf59" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44505 } }
I20260620 23:00:52.868036 20908 leader_election.cc:302] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f356a78465ed48b68b9f721ae862cf59; no voters:
I20260620 23:00:52.868228 20908 leader_election.cc:288] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:52.868332 20912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:52.868515 20912 raft_consensus.cc:686] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [term 1 LEADER]: Becoming Leader. State: Replica: f356a78465ed48b68b9f721ae862cf59, State: Running, Role: LEADER
I20260620 23:00:52.868530 20908 sys_catalog.cc:564] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:52.868656 20912 consensus_queue.cc:237] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356a78465ed48b68b9f721ae862cf59" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44505 } }
I20260620 23:00:52.869346 20914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f356a78465ed48b68b9f721ae862cf59. Latest consensus state: current_term: 1 leader_uuid: "f356a78465ed48b68b9f721ae862cf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356a78465ed48b68b9f721ae862cf59" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44505 } } }
I20260620 23:00:52.869438 20914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:52.869798 20915 sys_catalog.cc:455] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f356a78465ed48b68b9f721ae862cf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f356a78465ed48b68b9f721ae862cf59" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44505 } } }
I20260620 23:00:52.870510 20922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:52.870518 20915 sys_catalog.cc:458] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:52.871510 20922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:52.873591 20922 catalog_manager.cc:1347] Generated new cluster ID: 93134185a31047f192c3c022822d0e90
I20260620 23:00:52.873677 20922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:00:52 dist-test-slave-rdl9 krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:52.881579 20922 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:52.882287 20922 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:52.885370 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:52.868121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.1.166.62:44505)
0620 23:00:52.868321 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:52.868334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:52.868370 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:00:52.868513 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:52.868520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:52.868581 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:52.868763 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:52.869620 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:52.871679 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:52.872832 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:52.873446 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:52.877992 (+ 4546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:52.881260 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:52.881266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:52.881266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:52.881281 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:52.883395 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:52.883901 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:52.883905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:52.883905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:52.884025 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:52.884232 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:52.884232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:52.884233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:52.884554 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:52.884820 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:52.884939 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 23:00:52.885111 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260620 23:00:52.892803 20922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59: Generated new TSK 0
I20260620 23:00:52.893072 20922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
Entry for principal kudu/127.1.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab.
I20260620 23:00:52.925937 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44505
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:53.015494 20934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:53.015731 20934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:53.015758 20934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:53.017037 20934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:53.017107 20934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:53.017138 20934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 23:00:53.019505 20934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.1.keytab
--principal=kudu/127.1.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:53.019903 20934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:53.020234 20934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:53.026368 20939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:53.026499 20934 server_base.cc:1034] running on GCE node
W20260620 23:00:53.026397 20940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.026386 20942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:53.026830 20934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:53.027104 20934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:53.028311 20934 hybrid_clock.cc:648] HybridClock initialized: now 1781996453028295 us; error 28 us; skew 500 ppm
Jun 20 23:00:53 dist-test-slave-rdl9 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:53.031108 20934 init.cc:377] Logged in from keytab as kudu/127.1.166.1@KRBTEST.COM (short username kudu)
I20260620 23:00:53.031575 20934 webserver.cc:460] Webserver started at http://127.1.166.1:39571/ using document root <none> and password file <none>
I20260620 23:00:53.031819 20934 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:53.031870 20934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:53.031997 20934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:53.033088 20934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "76d9896053504c47bee75713d1b088e1"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "7391cdc9c0c36a8f96394267ecd07b5e"
server_key_iv: "782f0738d7f792597f01810fac3796ee"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.033414 20934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "76d9896053504c47bee75713d1b088e1"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "7391cdc9c0c36a8f96394267ecd07b5e"
server_key_iv: "782f0738d7f792597f01810fac3796ee"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.035040 20934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:00:53.036090 20949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.036351 20934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:53.036448 20934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "76d9896053504c47bee75713d1b088e1"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "7391cdc9c0c36a8f96394267ecd07b5e"
server_key_iv: "782f0738d7f792597f01810fac3796ee"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.036531 20934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:53.046891 20934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:53.049441 20934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:53.049623 20934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:53.049875 20934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:53.050192 20934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:53.050228 20934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.050259 20934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:53.050284 20934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.056614 20934 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:46257
I20260620 23:00:53.056690 21062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:46257 every 8 connection(s)
I20260620 23:00:53.057029 20934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 23:00:53.062800 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 20934
I20260620 23:00:53.062906 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 23:00:53.063112 1688 external_mini_cluster.cc:1428] Setting key 59bbe7e3eae940a5bc13684dc6fa5174
Jun 20 23:00:53 dist-test-slave-rdl9 krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.1@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:53.070804 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.057829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:36681 (local address 127.1.166.62:44505)
0620 23:00:53.057894 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.057896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.058111 (+ 215us) server_negotiation.cc:407] Connection header received
0620 23:00:53.058702 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.058703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.058716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.058769 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:53.059871 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.060616 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.062277 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.062581 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.062651 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.066045 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:53.066049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:53.066049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:53.066062 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:53.068238 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:53.069014 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:53.069016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:53.069016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:53.069093 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:53.069473 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:53.069474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:53.069474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:53.069636 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:53.069701 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.070426 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.070630 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:00:53.070976 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.057754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:44505 (local address 127.1.166.1:36681)
0620 23:00:53.057967 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.058013 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.058576 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.058870 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.058875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.059506 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:53.059746 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.059752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.060856 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.060859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.062095 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.062103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.062677 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.062678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.062834 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.063624 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:53.063634 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:53.065830 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:53.068432 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:53.068450 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:53.068454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:53.068890 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:53.069228 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:53.069229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:53.069230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:53.069410 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:53.069701 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:53.069710 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:53.070078 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.070379 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.070640 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260620 23:00:53.071833 21063 heartbeater.cc:347] Connected to a master server at 127.1.166.62:44505
I20260620 23:00:53.072014 21063 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:53.072301 21063 heartbeater.cc:511] Master 127.1.166.62:44505 requested a full tablet report, sending...
I20260620 23:00:53.073138 20872 ts_manager.cc:194] Registered new tserver with Master: 76d9896053504c47bee75713d1b088e1 (127.1.166.1:46257)
WARNING: no policy specified for kudu/127.1.166.2@KRBTEST.COM; defaulting to no policy
I20260620 23:00:53.073877 20872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.1@KRBTEST.COM'} at 127.1.166.1:36681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
Entry for principal kudu/127.1.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab.
I20260620 23:00:53.091560 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44505
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:53.183038 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:53.183317 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:53.183347 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:53.184962 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:53.185063 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:53.185101 21068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 23:00:53.187119 21068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.2.keytab
--principal=kudu/127.1.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:53.187456 21068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:53.187736 21068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:53.193287 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.193295 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.193269 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:53.193480 21068 server_base.cc:1034] running on GCE node
I20260620 23:00:53.193818 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:53.194168 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:53.195328 21068 hybrid_clock.cc:648] HybridClock initialized: now 1781996453195301 us; error 63 us; skew 500 ppm
Jun 20 23:00:53 dist-test-slave-rdl9 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:53.198088 21068 init.cc:377] Logged in from keytab as kudu/127.1.166.2@KRBTEST.COM (short username kudu)
I20260620 23:00:53.198660 21068 webserver.cc:460] Webserver started at http://127.1.166.2:36269/ using document root <none> and password file <none>
I20260620 23:00:53.198904 21068 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:53.198964 21068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:53.199208 21068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:53.200419 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "8847e17d3da94db5942fbdff88974357"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "8a0d46ee6c88197d57d0885d0a6d6dd1"
server_key_iv: "13d47f93648cc19c34373e2f469163c5"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.200783 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "8847e17d3da94db5942fbdff88974357"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "8a0d46ee6c88197d57d0885d0a6d6dd1"
server_key_iv: "13d47f93648cc19c34373e2f469163c5"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.202244 21068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:00:53.203402 21083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.203714 21068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:53.203855 21068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "8847e17d3da94db5942fbdff88974357"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "8a0d46ee6c88197d57d0885d0a6d6dd1"
server_key_iv: "13d47f93648cc19c34373e2f469163c5"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.203955 21068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:53.220026 21068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:53.222963 21068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:53.223151 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:53.223407 21068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:53.223847 21068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:53.223891 21068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.223920 21068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:53.223938 21068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.231750 21068 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:45145
I20260620 23:00:53.231881 21196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:45145 every 8 connection(s)
I20260620 23:00:53.232193 21068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 23:00:53.237624 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 21068
I20260620 23:00:53.237753 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 23:00:53.237927 1688 external_mini_cluster.cc:1428] Setting key a0276cc446a233577dfaa277204747fb
Jun 20 23:00:53 dist-test-slave-rdl9 krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.2@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:53.242645 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.232994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:52245 (local address 127.1.166.62:44505)
0620 23:00:53.233082 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.233084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.233255 (+ 171us) server_negotiation.cc:407] Connection header received
0620 23:00:53.233800 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.233802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.233814 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.233875 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:53.234950 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.235470 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.236651 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.236917 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.236975 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.239371 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:53.239373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:53.239373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:53.239382 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:53.240810 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:53.241351 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:53.241354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:53.241354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:53.241477 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:53.241859 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:53.241860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:53.241861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:53.242019 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:53.242117 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.242444 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.242545 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:00:53.242858 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.232948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:44505 (local address 127.1.166.2:52245)
0620 23:00:53.233123 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.233171 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.233688 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.233961 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.233965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.234610 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:53.234846 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.234852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.235565 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.235566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.236535 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.236538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.237000 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.237001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.237133 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.237700 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:53.237709 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:53.239277 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:53.240887 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:53.240896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:53.240898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:53.241193 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:53.241586 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:53.241587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:53.241587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:53.241741 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:53.242121 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:53.242130 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:53.242280 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.242419 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.242614 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260620 23:00:53.243340 21197 heartbeater.cc:347] Connected to a master server at 127.1.166.62:44505
I20260620 23:00:53.243443 21197 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:53.243799 21197 heartbeater.cc:511] Master 127.1.166.62:44505 requested a full tablet report, sending...
I20260620 23:00:53.244242 20872 ts_manager.cc:194] Registered new tserver with Master: 8847e17d3da94db5942fbdff88974357 (127.1.166.2:45145)
I20260620 23:00:53.244918 20872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.2@KRBTEST.COM'} at 127.1.166.2:52245
WARNING: no policy specified for kudu/127.1.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
Entry for principal kudu/127.1.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab.
I20260620 23:00:53.262702 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44505
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:53.350656 21202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:53.350914 21202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:53.350940 21202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:53.352232 21202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:53.352283 21202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:53.352313 21202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 23:00:53.353902 21202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.3.keytab
--principal=kudu/127.1.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:44505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:53.354136 21202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:53.354328 21202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:53.359639 21208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.359659 21210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.359654 21207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:53.359875 21202 server_base.cc:1034] running on GCE node
I20260620 23:00:53.360121 21202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:53.360406 21202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:53.361601 21202 hybrid_clock.cc:648] HybridClock initialized: now 1781996453361573 us; error 103 us; skew 500 ppm
Jun 20 23:00:53 dist-test-slave-rdl9 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:53.364521 21202 init.cc:377] Logged in from keytab as kudu/127.1.166.3@KRBTEST.COM (short username kudu)
I20260620 23:00:53.365058 21202 webserver.cc:460] Webserver started at http://127.1.166.3:34463/ using document root <none> and password file <none>
I20260620 23:00:53.365276 21202 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:53.365327 21202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:53.365442 21202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:53.366712 21202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "d92a4b7055354106904a08675655c0f9"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "90831ab7a7ecae6b2e7cea91ee3e1b35"
server_key_iv: "7c2b15fbadca726938e82eecc6b98143"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.367128 21202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "d92a4b7055354106904a08675655c0f9"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "90831ab7a7ecae6b2e7cea91ee3e1b35"
server_key_iv: "7c2b15fbadca726938e82eecc6b98143"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.368712 21202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:00:53.369653 21217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.369870 21202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:00:53.369966 21202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "d92a4b7055354106904a08675655c0f9"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "90831ab7a7ecae6b2e7cea91ee3e1b35"
server_key_iv: "7c2b15fbadca726938e82eecc6b98143"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.370069 21202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:53.387739 21202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:53.390264 21202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:53.390468 21202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:53.390712 21202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:53.391049 21202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:53.391088 21202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.391119 21202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:53.391135 21202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.397264 21202 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:38721
I20260620 23:00:53.397377 21330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:38721 every 8 connection(s)
I20260620 23:00:53.397665 21202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 23:00:53.398097 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 21202
I20260620 23:00:53.398195 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 23:00:53.398422 1688 external_mini_cluster.cc:1428] Setting key baa9309d8dc684410456c0bbc414311f
Jun 20 23:00:53 dist-test-slave-rdl9 krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996453, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.3@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:00:53.410715 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.398783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:35229 (local address 127.1.166.62:44505)
0620 23:00:53.398911 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.398913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.399445 (+ 532us) server_negotiation.cc:407] Connection header received
0620 23:00:53.400307 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.400308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.400319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.400380 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:53.401483 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.402130 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.403430 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.403659 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.403738 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.406944 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:53.406946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:53.406946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:53.406956 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:53.408473 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:53.409044 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:53.409046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:53.409047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:53.409128 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:53.409513 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:53.409515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:53.409515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:53.409859 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:53.409989 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.410496 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.410626 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 23:00:53.411144 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.398862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:44505 (local address 127.1.166.3:35229)
0620 23:00:53.399180 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.399317 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.400158 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.400445 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.400449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.401119 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:53.401354 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.401359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.402279 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.402280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.403263 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.403267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.403823 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.403825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.404004 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.404771 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:00:53.404782 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:53.406775 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:53.408614 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:53.408627 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:53.408630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:53.408940 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:53.409241 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:00:53.409242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:00:53.409242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:00:53.409389 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:00:53.410014 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:53.410022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:53.410269 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.410440 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.410678 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":135,"threads_started":1}
I20260620 23:00:53.411713 21331 heartbeater.cc:347] Connected to a master server at 127.1.166.62:44505
I20260620 23:00:53.411826 21331 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:53.412035 21331 heartbeater.cc:511] Master 127.1.166.62:44505 requested a full tablet report, sending...
I20260620 23:00:53.412559 20872 ts_manager.cc:194] Registered new tserver with Master: d92a4b7055354106904a08675655c0f9 (127.1.166.3:38721)
I20260620 23:00:53.413009 20872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.166.3@KRBTEST.COM'} at 127.1.166.3:35229
I20260620 23:00:53.422623 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:53.430972 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.424140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.1.166.62:44505)
0620 23:00:53.424207 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.424209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.424267 (+ 58us) server_negotiation.cc:407] Connection header received
0620 23:00:53.424396 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.424398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.424411 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.424490 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:53.425154 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.425660 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.426508 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.426850 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.426925 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.428137 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:53.428138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:53.428138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:53.428145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:53.429425 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:53.429926 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:53.429927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:53.429927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:53.429990 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:53.430236 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:53.430236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:53.430236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:53.430498 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:53.430593 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.430822 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.430902 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:00:53.432833 20872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:00:53.434542 20872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:53.441367 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.437189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45145 (local address 127.0.0.1:33776)
0620 23:00:53.437521 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.437532 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.437586 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.437860 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.437860 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.437862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:53.437892 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.437893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.438880 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.438881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.440206 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.440209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.441019 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.441020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.441118 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.441133 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.441201 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.441270 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":22,"thread_start_us":65,"threads_started":1}
I20260620 23:00:53.441473 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.437188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.1.166.1:46257)
0620 23:00:53.437573 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.437576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.437583 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:00:53.437616 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.437618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.437634 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.437733 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:53.437872 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.439103 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.440472 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.441141 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.441203 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.441250 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.441334 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.441381 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":75,"threads_started":1}
I20260620 23:00:53.441661 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.437105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.1.166.3:38721)
0620 23:00:53.437321 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.437326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.437441 (+ 115us) server_negotiation.cc:407] Connection header received
0620 23:00:53.437521 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.437524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.437541 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.437630 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:53.437942 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.439143 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.440274 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.441093 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.441164 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.441224 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.441531 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.441579 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260620 23:00:53.441481 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.437022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46257 (local address 127.0.0.1:55956)
0620 23:00:53.437366 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.437375 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.437433 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.437741 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.437742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.437745 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:53.437802 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.437804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.439204 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.439205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.440311 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.440365 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.441213 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.441214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.441269 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.441272 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.441320 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.441355 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":93,"threads_started":1}
I20260620 23:00:53.441810 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.436983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:38721 (local address 127.0.0.1:47076)
0620 23:00:53.437316 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.437357 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.437433 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.437638 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.437641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.437644 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:53.437729 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.437731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.439256 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.439258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.440206 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.440209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.441262 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.441263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.441330 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.441334 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.441377 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.441422 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":59,"threads_started":1}
I20260620 23:00:53.442144 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.437356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.1.166.2:45145)
0620 23:00:53.437707 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.437712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.437720 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:00:53.437753 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.437755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.437767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.437952 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:53.438013 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.438784 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.440384 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.440934 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.441003 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.441073 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.441991 (+ 918us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.442061 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":65,"threads_started":1}
I20260620 23:00:53.442605 20997 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae80a4fc3d24ab39f858121ac650997 (DEFAULT_TABLE table=test-table [id=acbd8024780240e980924d73183704bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:53.442658 21265 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae80a4fc3d24ab39f858121ac650997 (DEFAULT_TABLE table=test-table [id=acbd8024780240e980924d73183704bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:53.442730 21131 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae80a4fc3d24ab39f858121ac650997 (DEFAULT_TABLE table=test-table [id=acbd8024780240e980924d73183704bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:53.442906 20997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae80a4fc3d24ab39f858121ac650997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:53.442984 21265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae80a4fc3d24ab39f858121ac650997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:53.442983 21131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae80a4fc3d24ab39f858121ac650997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:53.445753 21348 tablet_bootstrap.cc:492] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1: Bootstrap starting.
I20260620 23:00:53.446192 21349 tablet_bootstrap.cc:492] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Bootstrap starting.
I20260620 23:00:53.446506 21348 tablet_bootstrap.cc:654] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:53.446478 21350 tablet_bootstrap.cc:492] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: Bootstrap starting.
I20260620 23:00:53.446852 21349 tablet_bootstrap.cc:654] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:53.447042 21348 log.cc:824] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:53.447077 21350 tablet_bootstrap.cc:654] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:53.447418 21349 log.cc:824] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:53.447595 21350 log.cc:824] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:53.447867 21348 tablet_bootstrap.cc:492] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1: No bootstrap required, opened a new log
I20260620 23:00:53.447937 21348 ts_tablet_manager.cc:1397] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:53.448199 21349 tablet_bootstrap.cc:492] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: No bootstrap required, opened a new log
I20260620 23:00:53.448256 21349 ts_tablet_manager.cc:1397] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:53.448513 21350 tablet_bootstrap.cc:492] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: No bootstrap required, opened a new log
I20260620 23:00:53.448635 21350 ts_tablet_manager.cc:1397] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:53.449610 21349 raft_consensus.cc:348] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.449764 21349 raft_consensus.cc:374] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:53.449786 21349 raft_consensus.cc:729] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8847e17d3da94db5942fbdff88974357, State: Initialized, Role: FOLLOWER
I20260620 23:00:53.449890 21349 consensus_queue.cc:260] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.449941 21348 raft_consensus.cc:348] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.450142 21348 raft_consensus.cc:374] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:53.450177 21349 ts_tablet_manager.cc:1428] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:53.450188 21348 raft_consensus.cc:729] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d9896053504c47bee75713d1b088e1, State: Initialized, Role: FOLLOWER
I20260620 23:00:53.450292 21197 heartbeater.cc:503] Master 127.1.166.62:44505 was elected leader, sending a full tablet report...
I20260620 23:00:53.450299 21348 consensus_queue.cc:260] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.450598 21348 ts_tablet_manager.cc:1428] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:53.450613 21063 heartbeater.cc:503] Master 127.1.166.62:44505 was elected leader, sending a full tablet report...
I20260620 23:00:53.451359 21350 raft_consensus.cc:348] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.451496 21350 raft_consensus.cc:374] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:53.451542 21350 raft_consensus.cc:729] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92a4b7055354106904a08675655c0f9, State: Initialized, Role: FOLLOWER
I20260620 23:00:53.451643 21350 consensus_queue.cc:260] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.451844 21350 ts_tablet_manager.cc:1428] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:00:53.451901 21331 heartbeater.cc:503] Master 127.1.166.62:44505 was elected leader, sending a full tablet report...
W20260620 23:00:53.482853 21198 tablet.cc:2369] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:00:53.517362 21356 raft_consensus.cc:482] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:53.517513 21356 raft_consensus.cc:504] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.517918 21356 leader_election.cc:288] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8847e17d3da94db5942fbdff88974357 (127.1.166.2:45145), 76d9896053504c47bee75713d1b088e1 (127.1.166.1:46257)
I20260620 23:00:53.520994 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.518096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:45145 (local address 127.1.166.3:38729)
0620 23:00:53.518175 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.518194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.518266 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.518535 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.518537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.518539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:53.518620 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.518623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.519298 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.519299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.520126 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.520129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.520711 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.520712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.520807 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.520811 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.520857 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.520914 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 23:00:53.520993 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.518275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:38729 (local address 127.1.166.2:45145)
0620 23:00:53.518359 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.518361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.518373 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:00:53.518402 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.518403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.518416 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.518499 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:53.518702 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.519213 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.520227 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.520630 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.520698 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.520711 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.520872 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.520915 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:00:53.521380 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.518490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:55009 (local address 127.1.166.1:46257)
0620 23:00:53.518554 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.518557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.518725 (+ 168us) server_negotiation.cc:407] Connection header received
0620 23:00:53.518764 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.518765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.518781 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.518825 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:53.519016 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.519529 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.520505 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.520946 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.521008 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.521023 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.521224 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.521282 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:00:53.521492 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae80a4fc3d24ab39f858121ac650997" candidate_uuid: "d92a4b7055354106904a08675655c0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8847e17d3da94db5942fbdff88974357" is_pre_election: true
I20260620 23:00:53.521657 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.518255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46257 (local address 127.1.166.3:55009)
0620 23:00:53.518589 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.518598 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.518665 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.518843 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.518844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.518846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:53.518951 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.518954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.519614 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.519615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.520403 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.520406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.521029 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.521030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.521110 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.521115 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.521516 (+ 401us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.521575 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":232,"threads_started":1}
I20260620 23:00:53.521853 21151 raft_consensus.cc:2454] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92a4b7055354106904a08675655c0f9 in term 0.
I20260620 23:00:53.522223 21219 leader_election.cc:302] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8847e17d3da94db5942fbdff88974357, d92a4b7055354106904a08675655c0f9; no voters:
I20260620 23:00:53.522248 21017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae80a4fc3d24ab39f858121ac650997" candidate_uuid: "d92a4b7055354106904a08675655c0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d9896053504c47bee75713d1b088e1" is_pre_election: true
I20260620 23:00:53.522480 21017 raft_consensus.cc:2454] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92a4b7055354106904a08675655c0f9 in term 0.
I20260620 23:00:53.522459 21356 raft_consensus.cc:2781] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:53.522576 21356 raft_consensus.cc:482] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:53.522615 21356 raft_consensus.cc:3037] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:53.523674 21356 raft_consensus.cc:504] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.523869 21356 leader_election.cc:288] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [CANDIDATE]: Term 1 election: Requested vote from peers 8847e17d3da94db5942fbdff88974357 (127.1.166.2:45145), 76d9896053504c47bee75713d1b088e1 (127.1.166.1:46257)
I20260620 23:00:53.524096 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae80a4fc3d24ab39f858121ac650997" candidate_uuid: "d92a4b7055354106904a08675655c0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8847e17d3da94db5942fbdff88974357"
I20260620 23:00:53.524196 21017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae80a4fc3d24ab39f858121ac650997" candidate_uuid: "d92a4b7055354106904a08675655c0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d9896053504c47bee75713d1b088e1"
I20260620 23:00:53.524236 21151 raft_consensus.cc:3037] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:53.524328 21017 raft_consensus.cc:3037] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:53.525249 21017 raft_consensus.cc:2454] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92a4b7055354106904a08675655c0f9 in term 1.
I20260620 23:00:53.525168 21151 raft_consensus.cc:2454] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92a4b7055354106904a08675655c0f9 in term 1.
I20260620 23:00:53.525496 21220 leader_election.cc:302] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d9896053504c47bee75713d1b088e1, d92a4b7055354106904a08675655c0f9; no voters:
I20260620 23:00:53.525622 21356 raft_consensus.cc:2781] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:53.525836 21356 raft_consensus.cc:686] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 1 LEADER]: Becoming Leader. State: Replica: d92a4b7055354106904a08675655c0f9, State: Running, Role: LEADER
I20260620 23:00:53.525944 21356 consensus_queue.cc:237] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } }
I20260620 23:00:53.526926 20872 catalog_manager.cc:5495] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d92a4b7055354106904a08675655c0f9 (127.1.166.3). New cstate: current_term: 1 leader_uuid: "d92a4b7055354106904a08675655c0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92a4b7055354106904a08675655c0f9" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:53.539193 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.536025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.1.166.3:38721)
0620 23:00:53.536156 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.536159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.536172 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:00:53.536227 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.536228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.536258 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.536376 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:00:53.536600 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.537106 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.537924 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.538128 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.538200 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.538520 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:00:53.538729 (+ 209us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:00:53.539008 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.539047 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.539077 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:00:53.542021 21151 raft_consensus.cc:1261] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 1 FOLLOWER]: Refusing update from remote peer d92a4b7055354106904a08675655c0f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:53.542021 21017 raft_consensus.cc:1261] T 2ae80a4fc3d24ab39f858121ac650997 P 76d9896053504c47bee75713d1b088e1 [term 1 FOLLOWER]: Refusing update from remote peer d92a4b7055354106904a08675655c0f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:53.542614 21358 consensus_queue.cc:1035] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d9896053504c47bee75713d1b088e1" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:53.542843 21356 consensus_queue.cc:1035] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8847e17d3da94db5942fbdff88974357" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 45145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:53.547744 21362 mvcc.cc:205] Tried to move back new op lower bound from 7299057473703702528 to 7299057473643200512. Current Snapshot: MvccSnapshot[applied={T|T < 7299057473703702528}]
I20260620 23:00:53.549857 20872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:53.550033 20872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:53.551204 20872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f356a78465ed48b68b9f721ae862cf59: Sending DeleteTablet for 3 replicas of tablet 2ae80a4fc3d24ab39f858121ac650997
I20260620 23:00:53.551652 21265 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae80a4fc3d24ab39f858121ac650997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:53 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:47076
I20260620 23:00:53.551666 20997 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae80a4fc3d24ab39f858121ac650997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:53 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:55956
I20260620 23:00:53.551946 21131 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae80a4fc3d24ab39f858121ac650997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:53 UTC) from {username='kudu', principal='kudu/127.1.166.62@KRBTEST.COM'} at 127.0.0.1:33776
I20260620 23:00:53.552073 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 20934
I20260620 23:00:53.552327 21372 tablet_replica.cc:330] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: stopping tablet replica
I20260620 23:00:53.552470 21372 raft_consensus.cc:2229] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:53.552974 21372 raft_consensus.cc:2258] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:53.553194 21370 tablet_replica.cc:330] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: stopping tablet replica
I20260620 23:00:53.553345 21370 raft_consensus.cc:2229] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:00:53.553706 21372 ts_tablet_manager.cc:1905] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:53.555624 21372 ts_tablet_manager.cc:1918] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:53.555702 21372 log.cc:1196] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/2ae80a4fc3d24ab39f858121ac650997
I20260620 23:00:53.556017 21372 ts_tablet_manager.cc:1939] T 2ae80a4fc3d24ab39f858121ac650997 P 8847e17d3da94db5942fbdff88974357: Deleting consensus metadata
I20260620 23:00:53.556507 21370 raft_consensus.cc:2258] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:53.556483 20857 catalog_manager.cc:4841] TS 8847e17d3da94db5942fbdff88974357 (127.1.166.2:45145): tablet 2ae80a4fc3d24ab39f858121ac650997 (table test-table [id=acbd8024780240e980924d73183704bf]) successfully deleted
I20260620 23:00:53.557054 21370 ts_tablet_manager.cc:1905] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:53.558846 21370 ts_tablet_manager.cc:1918] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:53.558938 21370 log.cc:1196] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/2ae80a4fc3d24ab39f858121ac650997
I20260620 23:00:53.559206 21370 ts_tablet_manager.cc:1939] T 2ae80a4fc3d24ab39f858121ac650997 P d92a4b7055354106904a08675655c0f9: Deleting consensus metadata
I20260620 23:00:53.559671 20857 catalog_manager.cc:4841] TS d92a4b7055354106904a08675655c0f9 (127.1.166.3:38721): tablet 2ae80a4fc3d24ab39f858121ac650997 (table test-table [id=acbd8024780240e980924d73183704bf]) successfully deleted
W20260620 23:00:53.559919 20858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:46257 (error 108)
I20260620 23:00:53.560483 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.560342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:46257 (local address 127.0.0.1:55962)
0620 23:00:53.560403 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.560446 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 23:00:53.560469 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 21068
W20260620 23:00:53.560635 20858 catalog_manager.cc:4584] TS 76d9896053504c47bee75713d1b088e1 (127.1.166.1:46257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ae80a4fc3d24ab39f858121ac650997: Network error: Client connection negotiation failed: client connection to 127.1.166.1:46257: connect: Connection refused (error 111)
I20260620 23:00:53.568313 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 21202
I20260620 23:00:53.573876 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 20840
2026-06-20T23:00:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1942 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T23:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:53Z Disabled control of system clock
I20260620 23:00:53.605127 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46625
--webserver_interface=127.1.166.62
--webserver_port=0
--builtin_ntp_servers=127.1.166.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:00:53.698673 21377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:53.699012 21377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:53.699040 21377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:53.700666 21377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:53.700758 21377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:53.700773 21377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:53.700786 21377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:53.700800 21377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:53.702906 21377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:46625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:53.703226 21377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:53.703521 21377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:00:53.708912 21377 server_base.cc:1034] running on GCE node
W20260620 23:00:53.708922 21383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.708922 21385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.709052 21382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:53.709363 21377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:53.709705 21377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:53.710898 21377 hybrid_clock.cc:648] HybridClock initialized: now 1781996453710873 us; error 36 us; skew 500 ppm
I20260620 23:00:53.712677 21377 webserver.cc:460] Webserver started at http://127.1.166.62:41907/ using document root <none> and password file <none>
I20260620 23:00:53.713040 21377 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:53.713104 21377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:53.713258 21377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:53.714671 21377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "7519c03dbc464fedbe076afd7e762cef"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "f6a17db1e93f58f8ce5d8099eab380e2"
server_key_iv: "99c1530508c1dfcb082ab13b8dd5ef94"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.715402 21377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "7519c03dbc464fedbe076afd7e762cef"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "f6a17db1e93f58f8ce5d8099eab380e2"
server_key_iv: "99c1530508c1dfcb082ab13b8dd5ef94"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.717037 21377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:00:53.718099 21391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.718502 21377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:53.718645 21377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "7519c03dbc464fedbe076afd7e762cef"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "f6a17db1e93f58f8ce5d8099eab380e2"
server_key_iv: "99c1530508c1dfcb082ab13b8dd5ef94"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.718792 21377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:53.735700 21377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:53.736076 21377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:53.736225 21377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:53.740573 21377 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:46625
I20260620 23:00:53.740620 21443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:46625 every 8 connection(s)
I20260620 23:00:53.740963 21377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:53.741963 21444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:53.744331 21444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef: Bootstrap starting.
I20260620 23:00:53.744889 21444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:53.745151 21444 log.cc:824] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:53.746016 21444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef: No bootstrap required, opened a new log
I20260620 23:00:53.748049 21444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7519c03dbc464fedbe076afd7e762cef" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46625 } }
I20260620 23:00:53.748219 21444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:53.748240 21444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7519c03dbc464fedbe076afd7e762cef, State: Initialized, Role: FOLLOWER
I20260620 23:00:53.748370 21444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7519c03dbc464fedbe076afd7e762cef" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46625 } }
I20260620 23:00:53.748452 21444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:53.748482 21444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:53.748520 21444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:53.749046 21444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7519c03dbc464fedbe076afd7e762cef" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46625 } }
I20260620 23:00:53.749145 21444 leader_election.cc:302] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7519c03dbc464fedbe076afd7e762cef; no voters:
I20260620 23:00:53.749287 21444 leader_election.cc:288] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:53.749429 21447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:53.749562 21444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:53.749735 21447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [term 1 LEADER]: Becoming Leader. State: Replica: 7519c03dbc464fedbe076afd7e762cef, State: Running, Role: LEADER
I20260620 23:00:53.749893 21447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7519c03dbc464fedbe076afd7e762cef" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46625 } }
I20260620 23:00:53.750593 21449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7519c03dbc464fedbe076afd7e762cef. Latest consensus state: current_term: 1 leader_uuid: "7519c03dbc464fedbe076afd7e762cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7519c03dbc464fedbe076afd7e762cef" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46625 } } }
I20260620 23:00:53.750710 21449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:53.750731 21448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7519c03dbc464fedbe076afd7e762cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7519c03dbc464fedbe076afd7e762cef" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 46625 } } }
I20260620 23:00:53.750811 21448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:53.750923 21460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:53.751214 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 21377
I20260620 23:00:53.751327 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:53.751472 21460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:53.751549 1688 external_mini_cluster.cc:1428] Setting key dc8b579bc31572d2e477aab3c099aac8
I20260620 23:00:53.753335 21460 catalog_manager.cc:1347] Generated new cluster ID: a218ab01e5e94ce18ac5eed5ec493038
I20260620 23:00:53.753394 21460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:53.757839 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.752069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.1.166.62:46625)
0620 23:00:53.752385 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.752394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.752428 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:00:53.752548 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.752556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.752645 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.752788 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:53.753134 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.754537 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.755616 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.756277 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.756406 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.757092 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:53.757097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:53.757097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:53.757325 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:53.757370 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:53.757386 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:53.757456 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.757520 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.757610 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":77,"threads_started":1}
I20260620 23:00:53.776247 21460 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:53.776803 21460 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:53.780087 21460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef: Generated new TSK 0
I20260620 23:00:53.780304 21460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:53.786499 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.1:0
--local_ip_for_outbound_sockets=127.1.166.1
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46625
--builtin_ntp_servers=127.1.166.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:00:53.877140 21468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:53.877374 21468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:53.877398 21468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:53.878932 21468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:53.878996 21468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:53.879026 21468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.1
I20260620 23:00:53.880635 21468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.166.1
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.1
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:53.880925 21468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:53.881253 21468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:00:53.886441 21468 server_base.cc:1034] running on GCE node
W20260620 23:00:53.886425 21473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.886425 21476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:53.886430 21474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:53.886886 21468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:53.887163 21468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:53.888345 21468 hybrid_clock.cc:648] HybridClock initialized: now 1781996453888326 us; error 30 us; skew 500 ppm
I20260620 23:00:53.890035 21468 webserver.cc:460] Webserver started at http://127.1.166.1:42941/ using document root <none> and password file <none>
I20260620 23:00:53.890429 21468 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:53.890497 21468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:53.890666 21468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:53.892005 21468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/data/instance:
uuid: "77b51402dcc242a8be798ce76f990770"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "4d39e406d04a5c2a2abc65f36d0507a4"
server_key_iv: "659ce5f323bd330b4529c530d49e8cc2"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.892400 21468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance:
uuid: "77b51402dcc242a8be798ce76f990770"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "4d39e406d04a5c2a2abc65f36d0507a4"
server_key_iv: "659ce5f323bd330b4529c530d49e8cc2"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.894078 21468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:00:53.895311 21482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.895555 21468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:53.895659 21468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/wal
uuid: "77b51402dcc242a8be798ce76f990770"
format_stamp: "Formatted at 2026-06-20 23:00:53 on dist-test-slave-rdl9"
server_key: "4d39e406d04a5c2a2abc65f36d0507a4"
server_key_iv: "659ce5f323bd330b4529c530d49e8cc2"
server_key_version: "encryptionkey@0"
I20260620 23:00:53.895761 21468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:53.909245 21468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:53.909613 21468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:53.910019 21468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:53.910305 21468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:53.910737 21468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:53.910779 21468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.910810 21468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:53.910828 21468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:53.916887 21468 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.1:45505
I20260620 23:00:53.916960 21595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.1:45505 every 8 connection(s)
I20260620 23:00:53.917311 21468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/data/info.pb
I20260620 23:00:53.922289 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 21468
I20260620 23:00:53.922474 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-0/wal/instance
I20260620 23:00:53.922758 1688 external_mini_cluster.cc:1428] Setting key 6713ce2cfa60760000964fd9472f2d8e
I20260620 23:00:53.924014 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.918135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:47287 (local address 127.1.166.62:46625)
0620 23:00:53.918196 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:53.918197 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:53.918411 (+ 214us) server_negotiation.cc:407] Connection header received
0620 23:00:53.919009 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:53.919010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:53.919025 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:53.919096 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:53.920232 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.920852 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.922135 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.922495 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.922585 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.923724 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:53.923725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:53.923726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:53.923731 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:53.923750 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:53.923754 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:53.923807 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:53.923912 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 23:00:53.923947 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:00:53.924057 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.2:0
--local_ip_for_outbound_sockets=127.1.166.2
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46625
--builtin_ntp_servers=127.1.166.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:00:53.924329 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:53.918056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46625 (local address 127.1.166.1:47287)
0620 23:00:53.918262 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:53.918306 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:53.918875 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:53.919312 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:53.919318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:53.919333 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:53.920001 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.920008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.920988 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.920989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.921951 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:53.921954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:53.922669 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:53.922672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:53.922881 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:53.923552 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:53.923553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:53.923591 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:53.923624 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:53.923825 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:53.923830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:53.923834 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:00:53.923917 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:00:53.924079 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260620 23:00:53.924883 21596 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46625
I20260620 23:00:53.924988 21596 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:53.925211 21596 heartbeater.cc:511] Master 127.1.166.62:46625 requested a full tablet report, sending...
I20260620 23:00:53.925874 21408 ts_manager.cc:194] Registered new tserver with Master: 77b51402dcc242a8be798ce76f990770 (127.1.166.1:45505)
I20260620 23:00:53.926808 21408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.1:47287
W20260620 23:00:54.012349 21599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:54.012603 21599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:54.012629 21599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:54.013876 21599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:54.013928 21599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:54.013957 21599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.2
I20260620 23:00:54.015543 21599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.166.2
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.2
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:54.015805 21599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:54.016088 21599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:54.021454 21604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:54.021459 21605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:54.021472 21607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:54.021656 21599 server_base.cc:1034] running on GCE node
I20260620 23:00:54.021886 21599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:54.022184 21599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:54.023360 21599 hybrid_clock.cc:648] HybridClock initialized: now 1781996454023320 us; error 51 us; skew 500 ppm
I20260620 23:00:54.025076 21599 webserver.cc:460] Webserver started at http://127.1.166.2:44793/ using document root <none> and password file <none>
I20260620 23:00:54.025343 21599 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:54.025413 21599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:54.025559 21599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:54.026870 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/data/instance:
uuid: "5f148226bd1a48d2b063bfde166bc17a"
format_stamp: "Formatted at 2026-06-20 23:00:54 on dist-test-slave-rdl9"
server_key: "0e1bc32cd657899aeb37d47caee03162"
server_key_iv: "05a836823a55b9b7c78e959ed93c5c7f"
server_key_version: "encryptionkey@0"
I20260620 23:00:54.027259 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance:
uuid: "5f148226bd1a48d2b063bfde166bc17a"
format_stamp: "Formatted at 2026-06-20 23:00:54 on dist-test-slave-rdl9"
server_key: "0e1bc32cd657899aeb37d47caee03162"
server_key_iv: "05a836823a55b9b7c78e959ed93c5c7f"
server_key_version: "encryptionkey@0"
I20260620 23:00:54.028795 21599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:00:54.029875 21613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:54.030162 21599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:54.030445 21599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/wal
uuid: "5f148226bd1a48d2b063bfde166bc17a"
format_stamp: "Formatted at 2026-06-20 23:00:54 on dist-test-slave-rdl9"
server_key: "0e1bc32cd657899aeb37d47caee03162"
server_key_iv: "05a836823a55b9b7c78e959ed93c5c7f"
server_key_version: "encryptionkey@0"
I20260620 23:00:54.030547 21599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:54.042711 21599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:54.043130 21599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:54.043283 21599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:54.043556 21599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:54.043979 21599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:54.044018 21599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:54.044051 21599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:54.044075 21599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:54.051817 21599 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.2:41439
I20260620 23:00:54.051916 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.2:41439 every 8 connection(s)
I20260620 23:00:54.052325 21599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/data/info.pb
I20260620 23:00:54.058584 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.053265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.2:55133 (local address 127.1.166.62:46625)
0620 23:00:54.053354 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.053357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.053524 (+ 167us) server_negotiation.cc:407] Connection header received
0620 23:00:54.054121 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.054122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.054133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.054193 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:54.055053 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.055569 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.056699 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.056915 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.056987 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.058074 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:54.058077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:54.058077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:54.058086 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:54.058113 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:54.058121 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:54.058298 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.058465 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.058505 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:00:54.058759 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.053175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46625 (local address 127.1.166.2:55133)
0620 23:00:54.053407 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:54.053449 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:54.053988 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:54.054258 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:54.054262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:54.054272 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:54.054919 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.054925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.055683 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.055684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.056552 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.056555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.057035 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.057036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.057190 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.057820 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:54.057821 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:54.057855 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:54.057886 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:54.058308 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:54.058316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:54.058320 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:00:54.058426 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:00:54.058536 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260620 23:00:54.059408 21727 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46625
I20260620 23:00:54.059520 21727 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:54.059692 21727 heartbeater.cc:511] Master 127.1.166.62:46625 requested a full tablet report, sending...
I20260620 23:00:54.060263 21408 ts_manager.cc:194] Registered new tserver with Master: 5f148226bd1a48d2b063bfde166bc17a (127.1.166.2:41439)
I20260620 23:00:54.060407 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 21599
I20260620 23:00:54.060524 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/wal/instance
I20260620 23:00:54.060772 1688 external_mini_cluster.cc:1428] Setting key 2431e906fc7da3b0c11dfe5684ca1b48
I20260620 23:00:54.060984 21408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.2:55133
I20260620 23:00:54.062281 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.166.3:0
--local_ip_for_outbound_sockets=127.1.166.3
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46625
--builtin_ntp_servers=127.1.166.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:00:54.154990 21730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:54.155234 21730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:54.155260 21730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:54.156780 21730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:54.156869 21730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:54.156912 21730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.166.3
I20260620 23:00:54.158610 21730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.166.3
--webserver_port=0
--tserver_master_addrs=127.1.166.62:46625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.166.3
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:54.158916 21730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:54.159286 21730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:54.164534 21736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:54.164534 21735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:54.164528 21738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:54.164961 21730 server_base.cc:1034] running on GCE node
I20260620 23:00:54.165277 21730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:54.165639 21730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:54.166822 21730 hybrid_clock.cc:648] HybridClock initialized: now 1781996454166774 us; error 55 us; skew 500 ppm
I20260620 23:00:54.168409 21730 webserver.cc:460] Webserver started at http://127.1.166.3:42943/ using document root <none> and password file <none>
I20260620 23:00:54.168689 21730 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:54.168744 21730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:54.168874 21730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:54.170287 21730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/data/instance:
uuid: "38df8460d2a24a6b9317aa1979adf108"
format_stamp: "Formatted at 2026-06-20 23:00:54 on dist-test-slave-rdl9"
server_key: "1ff2953c114c9070fee12b9bcfb09e72"
server_key_iv: "b238c1de3a5cbd612de5e569b7df2761"
server_key_version: "encryptionkey@0"
I20260620 23:00:54.170925 21730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance:
uuid: "38df8460d2a24a6b9317aa1979adf108"
format_stamp: "Formatted at 2026-06-20 23:00:54 on dist-test-slave-rdl9"
server_key: "1ff2953c114c9070fee12b9bcfb09e72"
server_key_iv: "b238c1de3a5cbd612de5e569b7df2761"
server_key_version: "encryptionkey@0"
I20260620 23:00:54.172721 21730 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 23:00:54.173887 21744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:54.174247 21730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:54.174433 21730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/wal
uuid: "38df8460d2a24a6b9317aa1979adf108"
format_stamp: "Formatted at 2026-06-20 23:00:54 on dist-test-slave-rdl9"
server_key: "1ff2953c114c9070fee12b9bcfb09e72"
server_key_iv: "b238c1de3a5cbd612de5e569b7df2761"
server_key_version: "encryptionkey@0"
I20260620 23:00:54.174540 21730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:54.188140 21730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:54.188489 21730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:54.188637 21730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:54.188865 21730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:54.189196 21730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:54.189231 21730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:54.189265 21730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:54.189285 21730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:54.195814 21730 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.3:45565
I20260620 23:00:54.195863 21857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.3:45565 every 8 connection(s)
I20260620 23:00:54.196236 21730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/data/info.pb
I20260620 23:00:54.198827 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 21730
I20260620 23:00:54.198966 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/wal/instance
I20260620 23:00:54.199183 1688 external_mini_cluster.cc:1428] Setting key 35d8bf163b66ba5ad4cb01b1e59ab458
I20260620 23:00:54.202172 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.196977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.3:48327 (local address 127.1.166.62:46625)
0620 23:00:54.197025 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.197027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.197272 (+ 245us) server_negotiation.cc:407] Connection header received
0620 23:00:54.197842 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.197844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.197861 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.197957 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:54.198757 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.199457 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.200719 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.200914 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.200971 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.201889 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:54.201890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:54.201891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:54.201895 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:54.201912 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:54.201917 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:54.201970 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.202082 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.202111 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:00:54.202323 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.196938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:46625 (local address 127.1.166.3:48327)
0620 23:00:54.197150 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:54.197186 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:54.197674 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:54.198012 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:54.198015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:54.198024 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:00:54.198602 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.198608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.199565 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.199566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.200643 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.200646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.201013 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.201015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.201146 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.201727 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:00:54.201729 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:00:54.201760 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:00:54.201787 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:00:54.201980 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:00:54.201983 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:00:54.201985 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:00:54.202039 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:00:54.202125 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260620 23:00:54.202910 21858 heartbeater.cc:347] Connected to a master server at 127.1.166.62:46625
I20260620 23:00:54.203013 21858 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:54.203212 21858 heartbeater.cc:511] Master 127.1.166.62:46625 requested a full tablet report, sending...
I20260620 23:00:54.203729 21408 ts_manager.cc:194] Registered new tserver with Master: 38df8460d2a24a6b9317aa1979adf108 (127.1.166.3:45565)
I20260620 23:00:54.204090 21408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.166.3:48327
I20260620 23:00:54.211927 1688 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:54.217338 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.213545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.1.166.62:46625)
0620 23:00:54.213666 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.213669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.213717 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:00:54.213875 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.213876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.213896 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.213972 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:54.214244 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.214872 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.215791 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.215999 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.216073 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.216962 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:54.216965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:54.216965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:54.216975 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:54.216996 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:54.217002 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:54.217062 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.217193 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.217226 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:00:54.219448 21408 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:00:54.221103 21408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:54.228256 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.223532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:45565 (local address 127.0.0.1:51214)
0620 23:00:54.223663 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:54.223707 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:54.223774 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:54.224242 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:54.224245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:54.224248 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:54.224352 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.224354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.225694 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.225695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.226830 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.226832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.227876 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.227877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.227988 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.228003 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:00:54.228087 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:00:54.228154 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":32,"thread_start_us":58,"threads_started":1}
I20260620 23:00:54.228435 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.223707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.1.166.1:45505)
0620 23:00:54.223924 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.223928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.224137 (+ 209us) server_negotiation.cc:407] Connection header received
0620 23:00:54.224177 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.224178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.224262 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.224333 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:54.224476 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.225287 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.226888 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.227681 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.227855 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.227990 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.228282 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.228347 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":122,"threads_started":1}
I20260620 23:00:54.228402 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.223905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.1.166.3:45565)
0620 23:00:54.224068 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.224071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.224081 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:00:54.224119 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.224121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.224137 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.224227 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:54.224433 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.225607 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.226896 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.227658 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.227746 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.227854 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.228180 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.228268 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260620 23:00:54.229110 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.223872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.1.166.2:41439)
0620 23:00:54.224022 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.224026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.224269 (+ 243us) server_negotiation.cc:407] Connection header received
0620 23:00:54.224311 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.224313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.224328 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.224396 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:54.224564 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.225681 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.226864 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.227667 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.227746 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.227855 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.228930 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.229011 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260620 23:00:54.228341 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.223542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:45505 (local address 127.0.0.1:38408)
0620 23:00:54.224065 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:54.224074 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:54.224129 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:54.224335 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:54.224336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:54.224338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:54.224396 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.224398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.225402 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.225403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.226780 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.226783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.228092 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.228093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.228147 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.228150 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:00:54.228198 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:00:54.228235 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":42,"thread_start_us":68,"threads_started":1}
I20260620 23:00:54.228883 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.223752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:41439 (local address 127.0.0.1:39696)
0620 23:00:54.224179 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:54.224186 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:54.224220 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:54.224458 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:54.224459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:54.224461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:54.224496 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.224497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.225739 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.225740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.226786 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.226789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.228463 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.228464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.228702 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.228707 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:00:54.228773 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:00:54.228819 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260620 23:00:54.229938 21792 tablet_service.cc:1467] Processing CreateTablet for tablet ab2b8cb891f841ccbade38146874293f (DEFAULT_TABLE table=test-table [id=f43535841e4741d09352ecf6bdac32e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:54.230147 21530 tablet_service.cc:1467] Processing CreateTablet for tablet ab2b8cb891f841ccbade38146874293f (DEFAULT_TABLE table=test-table [id=f43535841e4741d09352ecf6bdac32e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:54.230487 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2b8cb891f841ccbade38146874293f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:54.230681 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2b8cb891f841ccbade38146874293f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:54.231209 21661 tablet_service.cc:1467] Processing CreateTablet for tablet ab2b8cb891f841ccbade38146874293f (DEFAULT_TABLE table=test-table [id=f43535841e4741d09352ecf6bdac32e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:54.231577 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2b8cb891f841ccbade38146874293f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:54.234560 21872 tablet_bootstrap.cc:492] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770: Bootstrap starting.
I20260620 23:00:54.235281 21872 tablet_bootstrap.cc:654] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:54.235565 21873 tablet_bootstrap.cc:492] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: Bootstrap starting.
I20260620 23:00:54.235601 21872 log.cc:824] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:54.236294 21873 tablet_bootstrap.cc:654] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:54.236516 21872 tablet_bootstrap.cc:492] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770: No bootstrap required, opened a new log
I20260620 23:00:54.236588 21872 ts_tablet_manager.cc:1397] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:00:54.236621 21873 log.cc:824] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:54.237704 21876 tablet_bootstrap.cc:492] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Bootstrap starting.
I20260620 23:00:54.238435 21876 tablet_bootstrap.cc:654] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:54.238773 21876 log.cc:824] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:54.238988 21873 tablet_bootstrap.cc:492] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: No bootstrap required, opened a new log
I20260620 23:00:54.239071 21873 ts_tablet_manager.cc:1397] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 23:00:54.239029 21872 raft_consensus.cc:348] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.239212 21872 raft_consensus.cc:374] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:54.239238 21872 raft_consensus.cc:729] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b51402dcc242a8be798ce76f990770, State: Initialized, Role: FOLLOWER
I20260620 23:00:54.239331 21872 consensus_queue.cc:260] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.239578 21872 ts_tablet_manager.cc:1428] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:54.239864 21876 tablet_bootstrap.cc:492] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: No bootstrap required, opened a new log
I20260620 23:00:54.239919 21876 ts_tablet_manager.cc:1397] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:00:54.241186 21873 raft_consensus.cc:348] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.241343 21873 raft_consensus.cc:374] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:54.241375 21873 raft_consensus.cc:729] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38df8460d2a24a6b9317aa1979adf108, State: Initialized, Role: FOLLOWER
I20260620 23:00:54.241497 21873 consensus_queue.cc:260] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.241732 21873 ts_tablet_manager.cc:1428] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:54.241809 21858 heartbeater.cc:503] Master 127.1.166.62:46625 was elected leader, sending a full tablet report...
I20260620 23:00:54.241832 21876 raft_consensus.cc:348] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.241943 21876 raft_consensus.cc:374] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:54.241961 21876 raft_consensus.cc:729] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f148226bd1a48d2b063bfde166bc17a, State: Initialized, Role: FOLLOWER
I20260620 23:00:54.242044 21876 consensus_queue.cc:260] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.242266 21876 ts_tablet_manager.cc:1428] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:00:54.242426 21727 heartbeater.cc:503] Master 127.1.166.62:46625 was elected leader, sending a full tablet report...
I20260620 23:00:54.242477 21596 heartbeater.cc:503] Master 127.1.166.62:46625 was elected leader, sending a full tablet report...
W20260620 23:00:54.302999 21728 tablet.cc:2369] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:00:54.340214 21878 raft_consensus.cc:482] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:54.340363 21878 raft_consensus.cc:504] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.340773 21878 leader_election.cc:288] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38df8460d2a24a6b9317aa1979adf108 (127.1.166.3:45565), 5f148226bd1a48d2b063bfde166bc17a (127.1.166.2:41439)
I20260620 23:00:54.344731 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.341042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.3:45565 (local address 127.1.166.1:40451)
0620 23:00:54.341101 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:54.341113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:54.341180 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:54.341378 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:54.341378 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:54.341380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:54.341452 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.341454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.342520 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.342522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.343452 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.343455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.344372 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.344374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.344467 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.344473 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:00:54.344552 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:00:54.344615 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 23:00:54.344770 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.341127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:40451 (local address 127.1.166.3:45565)
0620 23:00:54.341249 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.341251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.341260 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:00:54.341294 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.341295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.341306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.341364 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:54.341604 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.342389 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.343589 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.344163 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.344392 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.344416 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.344570 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.344698 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 23:00:54.345268 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2b8cb891f841ccbade38146874293f" candidate_uuid: "77b51402dcc242a8be798ce76f990770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38df8460d2a24a6b9317aa1979adf108" is_pre_election: true
I20260620 23:00:54.345414 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.341142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.2:41439 (local address 127.1.166.1:56527)
0620 23:00:54.341315 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:54.341392 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 23:00:54.341535 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:00:54.341742 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:00:54.341742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:00:54.341745 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:00:54.341854 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.341857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.342807 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.342808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.343942 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:00:54.343947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.345047 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.345048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:00:54.345164 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.345173 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:00:54.345258 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:00:54.345320 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
I20260620 23:00:54.345477 21812 raft_consensus.cc:2454] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b51402dcc242a8be798ce76f990770 in term 0.
I20260620 23:00:54.345551 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.341232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.166.1:56527 (local address 127.1.166.2:41439)
0620 23:00:54.341285 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.341287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.341511 (+ 224us) server_negotiation.cc:407] Connection header received
0620 23:00:54.341605 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.341606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.341616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.341669 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:00:54.342004 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.342673 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.344124 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.344699 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.344815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.344830 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.345383 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.345462 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:00:54.345778 21486 leader_election.cc:302] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38df8460d2a24a6b9317aa1979adf108, 77b51402dcc242a8be798ce76f990770; no voters:
I20260620 23:00:54.345953 21878 raft_consensus.cc:2781] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:54.345999 21878 raft_consensus.cc:482] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:54.346027 21878 raft_consensus.cc:3037] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:54.345986 21681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2b8cb891f841ccbade38146874293f" candidate_uuid: "77b51402dcc242a8be798ce76f990770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f148226bd1a48d2b063bfde166bc17a" is_pre_election: true
I20260620 23:00:54.346200 21681 raft_consensus.cc:2454] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77b51402dcc242a8be798ce76f990770 in term 0.
I20260620 23:00:54.346951 21878 raft_consensus.cc:504] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.347143 21878 leader_election.cc:288] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [CANDIDATE]: Term 1 election: Requested vote from peers 38df8460d2a24a6b9317aa1979adf108 (127.1.166.3:45565), 5f148226bd1a48d2b063bfde166bc17a (127.1.166.2:41439)
I20260620 23:00:54.347481 21812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2b8cb891f841ccbade38146874293f" candidate_uuid: "77b51402dcc242a8be798ce76f990770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38df8460d2a24a6b9317aa1979adf108"
I20260620 23:00:54.347517 21681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2b8cb891f841ccbade38146874293f" candidate_uuid: "77b51402dcc242a8be798ce76f990770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f148226bd1a48d2b063bfde166bc17a"
I20260620 23:00:54.347600 21812 raft_consensus.cc:3037] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:54.347618 21681 raft_consensus.cc:3037] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:54.348642 21812 raft_consensus.cc:2454] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b51402dcc242a8be798ce76f990770 in term 1.
I20260620 23:00:54.348664 21681 raft_consensus.cc:2454] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77b51402dcc242a8be798ce76f990770 in term 1.
I20260620 23:00:54.348896 21486 leader_election.cc:302] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f148226bd1a48d2b063bfde166bc17a, 77b51402dcc242a8be798ce76f990770; no voters:
I20260620 23:00:54.349020 21878 raft_consensus.cc:2781] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:54.349192 21878 raft_consensus.cc:686] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [term 1 LEADER]: Becoming Leader. State: Replica: 77b51402dcc242a8be798ce76f990770, State: Running, Role: LEADER
I20260620 23:00:54.349300 21878 consensus_queue.cc:237] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } }
I20260620 23:00:54.350627 21408 catalog_manager.cc:5495] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77b51402dcc242a8be798ce76f990770 (127.1.166.1). New cstate: current_term: 1 leader_uuid: "77b51402dcc242a8be798ce76f990770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77b51402dcc242a8be798ce76f990770" member_type: VOTER last_known_addr { host: "127.1.166.1" port: 45505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:54.384348 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.381457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.1.166.1:45505)
0620 23:00:54.381551 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:54.381553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:54.381564 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:00:54.381608 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:54.381609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:54.381633 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:54.381733 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:00:54.381898 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.382559 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.383450 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:54.383693 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:54.383768 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:54.383947 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:00:54.384073 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:00:54.384137 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:54.384217 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 23:00:54.384238 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:00:54.386548 21812 raft_consensus.cc:1261] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 1 FOLLOWER]: Refusing update from remote peer 77b51402dcc242a8be798ce76f990770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:54.386548 21681 raft_consensus.cc:1261] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 1 FOLLOWER]: Refusing update from remote peer 77b51402dcc242a8be798ce76f990770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:54.386971 21882 consensus_queue.cc:1035] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f148226bd1a48d2b063bfde166bc17a" member_type: VOTER last_known_addr { host: "127.1.166.2" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:54.387130 21878 consensus_queue.cc:1035] T ab2b8cb891f841ccbade38146874293f P 77b51402dcc242a8be798ce76f990770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38df8460d2a24a6b9317aa1979adf108" member_type: VOTER last_known_addr { host: "127.1.166.3" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:54.390506 21886 mvcc.cc:205] Tried to move back new op lower bound from 7299057477164101632 to 7299057477015609344. Current Snapshot: MvccSnapshot[applied={T|T < 7299057477164101632}]
I20260620 23:00:54.390887 21887 mvcc.cc:205] Tried to move back new op lower bound from 7299057477164101632 to 7299057477015609344. Current Snapshot: MvccSnapshot[applied={T|T < 7299057477164101632 or (T in {7299057477164101632})}]
I20260620 23:00:54.394439 21408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:54.394632 21408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:00:54.396102 21408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7519c03dbc464fedbe076afd7e762cef: Sending DeleteTablet for 3 replicas of tablet ab2b8cb891f841ccbade38146874293f
I20260620 23:00:54.396605 21792 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2b8cb891f841ccbade38146874293f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:54 UTC) from {username='slave'} at 127.0.0.1:51214
I20260620 23:00:54.396638 21661 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2b8cb891f841ccbade38146874293f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:54 UTC) from {username='slave'} at 127.0.0.1:39696
I20260620 23:00:54.396935 21896 tablet_replica.cc:330] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: stopping tablet replica
I20260620 23:00:54.396945 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2b8cb891f841ccbade38146874293f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:00:54 UTC) from {username='slave'} at 127.0.0.1:38408
I20260620 23:00:54.397075 21896 raft_consensus.cc:2229] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:54.397161 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 21468
I20260620 23:00:54.397207 21896 raft_consensus.cc:2258] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:54.397364 21895 tablet_replica.cc:330] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: stopping tablet replica
I20260620 23:00:54.397467 21895 raft_consensus.cc:2229] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:00:54.397547 21895 raft_consensus.cc:2258] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:00:54.398221 21896 ts_tablet_manager.cc:1905] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:54.398221 21895 ts_tablet_manager.cc:1905] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:00:54.399993 21895 ts_tablet_manager.cc:1918] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:54.399994 21896 ts_tablet_manager.cc:1918] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:00:54.400091 21895 log.cc:1196] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-1/wal/wals/ab2b8cb891f841ccbade38146874293f
I20260620 23:00:54.400091 21896 log.cc:1196] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: Deleting WAL directory at /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996175763767-1688-0/minicluster-data/ts-2/wal/wals/ab2b8cb891f841ccbade38146874293f
I20260620 23:00:54.400358 21896 ts_tablet_manager.cc:1939] T ab2b8cb891f841ccbade38146874293f P 38df8460d2a24a6b9317aa1979adf108: Deleting consensus metadata
I20260620 23:00:54.400355 21895 ts_tablet_manager.cc:1939] T ab2b8cb891f841ccbade38146874293f P 5f148226bd1a48d2b063bfde166bc17a: Deleting consensus metadata
I20260620 23:00:54.400902 21395 catalog_manager.cc:4841] TS 5f148226bd1a48d2b063bfde166bc17a (127.1.166.2:41439): tablet ab2b8cb891f841ccbade38146874293f (table test-table [id=f43535841e4741d09352ecf6bdac32e2]) successfully deleted
I20260620 23:00:54.401042 21395 catalog_manager.cc:4841] TS 38df8460d2a24a6b9317aa1979adf108 (127.1.166.3:45565): tablet ab2b8cb891f841ccbade38146874293f (table test-table [id=f43535841e4741d09352ecf6bdac32e2]) successfully deleted
W20260620 23:00:54.403456 21392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.166.1:45505 (error 108)
I20260620 23:00:54.403636 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 21599
I20260620 23:00:54.404528 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:54.404078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.1:45505 (local address 127.0.0.1:38416)
0620 23:00:54.404422 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 23:00:54.404480 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318}
W20260620 23:00:54.408159 21392 catalog_manager.cc:4584] TS 77b51402dcc242a8be798ce76f990770 (127.1.166.1:45505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab2b8cb891f841ccbade38146874293f: Network error: Client connection negotiation failed: client connection to 127.1.166.1:45505: connect: Connection refused (error 111)
I20260620 23:00:54.408286 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 21730
I20260620 23:00:54.413209 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 21377
2026-06-20T23:00:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (838 ms)
[----------] 6 tests from EncryptionPolicyTest (8831 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:00:54 dist-test-slave-rdl9 krb5kdc[21901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:00:54 dist-test-slave-rdl9 krb5kdc[21901](info): set up 2 sockets
Jun 20 23:00:54 dist-test-slave-rdl9 krb5kdc[21901](info): commencing operation
krb5kdc: starting...
W20260620 23:00:55.390873 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:00:55 dist-test-slave-rdl9 krb5kdc[21901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:00:55.521128 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:55.614152 21914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:55.614504 21914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:55.614557 21914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:55.616158 21914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:55.616235 21914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:55.616256 21914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:55.616279 21914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:55.616295 21914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:55.618153 21914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:55.618515 21914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:55.618773 21914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:55.623770 21919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:55.623773 21922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:55.623773 21920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:55.624315 21914 server_base.cc:1034] running on GCE node
I20260620 23:00:55.624574 21914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:55.624904 21914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:55.626091 21914 hybrid_clock.cc:648] HybridClock initialized: now 1781996455626062 us; error 53 us; skew 500 ppm
Jun 20 23:00:55 dist-test-slave-rdl9 krb5kdc[21901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:55.628980 21914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:00:55.629526 21914 webserver.cc:460] Webserver started at http://127.0.0.1:35407/ using document root <none> and password file <none>
I20260620 23:00:55.629873 21914 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:55.629937 21914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:55.630053 21914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:55.631441 21914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "8e369c62c9404dd1b72752b02498d7ab"
format_stamp: "Formatted at 2026-06-20 23:00:55 on dist-test-slave-rdl9"
server_key: "3eea0a847f401082ff7738a4d5184c68"
server_key_iv: "b18b9c31b199dfed7c6c7ca3f72a840c"
server_key_version: "encryptionkey@0"
I20260620 23:00:55.631771 21914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "8e369c62c9404dd1b72752b02498d7ab"
format_stamp: "Formatted at 2026-06-20 23:00:55 on dist-test-slave-rdl9"
server_key: "3eea0a847f401082ff7738a4d5184c68"
server_key_iv: "b18b9c31b199dfed7c6c7ca3f72a840c"
server_key_version: "encryptionkey@0"
I20260620 23:00:55.633154 21914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:00:55.634251 21929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:55.634516 21914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:00:55.634666 21914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "8e369c62c9404dd1b72752b02498d7ab"
format_stamp: "Formatted at 2026-06-20 23:00:55 on dist-test-slave-rdl9"
server_key: "3eea0a847f401082ff7738a4d5184c68"
server_key_iv: "b18b9c31b199dfed7c6c7ca3f72a840c"
server_key_version: "encryptionkey@0"
I20260620 23:00:55.634770 21914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:55.653910 21914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:55.656262 21914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:55.656387 21914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:55.660955 21914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46319
I20260620 23:00:55.660990 21981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46319 every 8 connection(s)
I20260620 23:00:55.661347 21914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:55.662528 21982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:55.664811 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab: Bootstrap starting.
I20260620 23:00:55.665355 21982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:55.665596 21982 log.cc:824] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:55.666383 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab: No bootstrap required, opened a new log
I20260620 23:00:55.667145 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 21914
I20260620 23:00:55.667234 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:55.667416 1688 external_mini_cluster.cc:1428] Setting key 14c020ae556a3aa8d55d128eff326642
I20260620 23:00:55.668321 21982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e369c62c9404dd1b72752b02498d7ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } }
I20260620 23:00:55.668478 21982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:55.668532 21982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e369c62c9404dd1b72752b02498d7ab, State: Initialized, Role: FOLLOWER
I20260620 23:00:55.668663 21982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e369c62c9404dd1b72752b02498d7ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } }
I20260620 23:00:55.668751 21982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:55.668781 21982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:55.668823 21982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:55.669870 21982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e369c62c9404dd1b72752b02498d7ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } }
I20260620 23:00:55.669988 21982 leader_election.cc:302] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e369c62c9404dd1b72752b02498d7ab; no voters:
I20260620 23:00:55.670161 21982 leader_election.cc:288] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:55.670249 21987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:55.670477 21987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [term 1 LEADER]: Becoming Leader. State: Replica: 8e369c62c9404dd1b72752b02498d7ab, State: Running, Role: LEADER
I20260620 23:00:55.670478 21982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:55.670680 21987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e369c62c9404dd1b72752b02498d7ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } }
I20260620 23:00:55.671444 21989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e369c62c9404dd1b72752b02498d7ab. Latest consensus state: current_term: 1 leader_uuid: "8e369c62c9404dd1b72752b02498d7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e369c62c9404dd1b72752b02498d7ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } } }
I20260620 23:00:55.671537 21989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:55.671646 21988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e369c62c9404dd1b72752b02498d7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e369c62c9404dd1b72752b02498d7ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46319 } } }
I20260620 23:00:55.671729 21988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab [sys.catalog]: This master's current role is: LEADER
W20260620 23:00:55.672619 22003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:00:55.672749 22003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:00:55.673023 21998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:55.673288 21998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:00:55 dist-test-slave-rdl9 krb5kdc[21901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:00:55.675474 21998 catalog_manager.cc:1347] Generated new cluster ID: af2fb3d9d42f4762bbbff87227769de7
I20260620 23:00:55.675526 21998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:55.682607 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:55.668003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43785 (local address 127.0.0.1:46319)
0620 23:00:55.668186 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:55.668195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:55.668223 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:00:55.668328 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:55.668333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:55.668378 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:55.668507 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:55.669322 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:55.670444 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:55.671795 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:55.672346 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:55.672449 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:55.675073 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:55.675078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:55.675079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:55.675092 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:55.680762 (+ 5670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:55.681333 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:55.681336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:55.681336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:55.681453 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:55.681680 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:55.681681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:55.681681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:55.681974 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:55.682126 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:55.682251 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 23:00:55.682375 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260620 23:00:55.683766 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:00:55.689450 21998 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:55.689975 21998 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:55.696228 21998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e369c62c9404dd1b72752b02498d7ab: Generated new TSK 0
I20260620 23:00:55.696460 21998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:55.701715 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:55.686626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46867 (local address 127.0.0.1:46319)
0620 23:00:55.686680 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:55.686682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:55.689073 (+ 2391us) server_negotiation.cc:407] Connection header received
0620 23:00:55.689132 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:55.689133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:55.689143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:55.689188 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:55.691009 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:55.691748 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:55.692841 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:55.693169 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:55.693236 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:55.699021 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:55.699023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:55.699024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:55.699033 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:55.700441 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:55.700971 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:55.700973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:55.700973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:55.701032 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:55.701246 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:55.701247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:55.701247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:55.701356 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:55.701442 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:55.701591 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:00:55.701645 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:00:55.704715 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 21914
W20260620 23:00:55.709333 21899 connection.cc:535] client connection to 127.0.0.1:46319 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46319): Connection reset by peer (error 104)
2026-06-20T23:00:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:00:55 dist-test-slave-rdl9 krb5kdc[22013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:00:55 dist-test-slave-rdl9 krb5kdc[22013](info): set up 2 sockets
Jun 20 23:00:55 dist-test-slave-rdl9 krb5kdc[22013](info): commencing operation
krb5kdc: starting...
W20260620 23:00:56.690868 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:00:56 dist-test-slave-rdl9 krb5kdc[22013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:00:56.821427 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:56.907972 22026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:56.908228 22026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:56.908255 22026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:56.909569 22026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:56.909632 22026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:56.909646 22026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:56.909658 22026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:56.909669 22026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:56.911402 22026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:56.911660 22026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:56.911878 22026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:56.917112 22031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:56.917132 22032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:56.917297 22026 server_base.cc:1034] running on GCE node
W20260620 23:00:56.917132 22034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:56.917656 22026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:56.918032 22026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:56.919227 22026 hybrid_clock.cc:648] HybridClock initialized: now 1781996456919200 us; error 41 us; skew 500 ppm
Jun 20 23:00:56 dist-test-slave-rdl9 krb5kdc[22013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:56.922005 22026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:00:56.922540 22026 webserver.cc:460] Webserver started at http://127.0.0.1:36231/ using document root <none> and password file <none>
I20260620 23:00:56.922809 22026 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:56.922863 22026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:56.922988 22026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:56.924165 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "ce9af834498f4eafaf13966a3e82faca"
format_stamp: "Formatted at 2026-06-20 23:00:56 on dist-test-slave-rdl9"
server_key: "a63dea8c24ae5dc7bf6916f9d1f95680"
server_key_iv: "f28f26e1a1b8fa5165879064bf485f36"
server_key_version: "encryptionkey@0"
I20260620 23:00:56.924512 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "ce9af834498f4eafaf13966a3e82faca"
format_stamp: "Formatted at 2026-06-20 23:00:56 on dist-test-slave-rdl9"
server_key: "a63dea8c24ae5dc7bf6916f9d1f95680"
server_key_iv: "f28f26e1a1b8fa5165879064bf485f36"
server_key_version: "encryptionkey@0"
I20260620 23:00:56.925981 22026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:00:56.927107 22041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:56.927380 22026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:00:56.927487 22026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "ce9af834498f4eafaf13966a3e82faca"
format_stamp: "Formatted at 2026-06-20 23:00:56 on dist-test-slave-rdl9"
server_key: "a63dea8c24ae5dc7bf6916f9d1f95680"
server_key_iv: "f28f26e1a1b8fa5165879064bf485f36"
server_key_version: "encryptionkey@0"
I20260620 23:00:56.927582 22026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:56.942162 22026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:56.944619 22026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:56.944757 22026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:56.949049 22026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40597
I20260620 23:00:56.949110 22093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40597 every 8 connection(s)
I20260620 23:00:56.949479 22026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:56.950640 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:56.953220 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca: Bootstrap starting.
I20260620 23:00:56.953891 22094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:56.954180 22094 log.cc:824] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:56.955201 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca: No bootstrap required, opened a new log
I20260620 23:00:56.957232 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22026
I20260620 23:00:56.957042 22094 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9af834498f4eafaf13966a3e82faca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40597 } }
I20260620 23:00:56.957304 22094 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:56.957350 22094 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9af834498f4eafaf13966a3e82faca, State: Initialized, Role: FOLLOWER
I20260620 23:00:56.957368 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:56.957571 1688 external_mini_cluster.cc:1428] Setting key 8c17c0a60e8477ed95433cd3fbd37caa
I20260620 23:00:56.957532 22094 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9af834498f4eafaf13966a3e82faca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40597 } }
I20260620 23:00:56.957615 22094 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:56.957660 22094 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:56.957734 22094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:56.958999 22094 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9af834498f4eafaf13966a3e82faca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40597 } }
I20260620 23:00:56.959136 22094 leader_election.cc:302] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce9af834498f4eafaf13966a3e82faca; no voters:
I20260620 23:00:56.959327 22094 leader_election.cc:288] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:56.959398 22099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:56.959627 22099 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [term 1 LEADER]: Becoming Leader. State: Replica: ce9af834498f4eafaf13966a3e82faca, State: Running, Role: LEADER
I20260620 23:00:56.959627 22094 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:56.959836 22099 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9af834498f4eafaf13966a3e82faca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40597 } }
I20260620 23:00:56.960282 22100 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce9af834498f4eafaf13966a3e82faca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9af834498f4eafaf13966a3e82faca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40597 } } }
I20260620 23:00:56.960366 22100 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:56.960577 22101 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce9af834498f4eafaf13966a3e82faca. Latest consensus state: current_term: 1 leader_uuid: "ce9af834498f4eafaf13966a3e82faca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce9af834498f4eafaf13966a3e82faca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40597 } } }
I20260620 23:00:56.960693 22101 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:56.961040 22108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:56.961710 22108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:56.963729 22108 catalog_manager.cc:1347] Generated new cluster ID: 9c27eade0a6a48a18aab450fbfa1ac9c
I20260620 23:00:56.963809 22108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:56.964049 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:56.958134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44053 (local address 127.0.0.1:40597)
0620 23:00:56.958369 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:56.958381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:56.958421 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:00:56.958551 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:56.958565 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:56.958613 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:56.958741 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:56.958987 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:56.960337 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:56.961571 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:56.962158 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:56.962273 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:56.963181 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:56.963187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:56.963187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:56.963380 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:56.963417 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:56.963431 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:56.963513 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:56.963788 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:00:56.963823 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260620 23:00:56.965180 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:00:56.975026 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:56.968245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44931 (local address 127.0.0.1:40597)
0620 23:00:56.968289 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:56.968291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:56.968452 (+ 161us) server_negotiation.cc:407] Connection header received
0620 23:00:56.968563 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:56.968564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:56.968576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:56.968606 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:56.968822 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:56.970024 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:56.971227 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:56.971594 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:56.971636 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:56.974731 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:56.974733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:56.974733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:56.974740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:56.974760 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:56.974766 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:56.974799 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:56.974912 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:00:56.974931 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:00:56.991173 22108 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:56.991931 22108 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:57.008050 22108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce9af834498f4eafaf13966a3e82faca: Generated new TSK 0
I20260620 23:00:57.008297 22108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:57.016505 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22026
W20260620 23:00:57.021428 22011 connection.cc:535] client connection to 127.0.0.1:40597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40597): Connection reset by peer (error 104)
2026-06-20T23:00:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:00:57 dist-test-slave-rdl9 krb5kdc[22125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:00:57 dist-test-slave-rdl9 krb5kdc[22125](info): set up 2 sockets
Jun 20 23:00:57 dist-test-slave-rdl9 krb5kdc[22125](info): commencing operation
krb5kdc: starting...
W20260620 23:00:57.998379 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:00:58 dist-test-slave-rdl9 krb5kdc[22125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:00:58.125106 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:58.219306 22138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:58.219595 22138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:58.219628 22138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:58.221513 22138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:58.221606 22138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:58.221624 22138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:58.221639 22138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:58.221653 22138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:58.223879 22138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:58.224205 22138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:58.224439 22138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:58.229895 22143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:58.230062 22146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:58.229887 22144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:58.230432 22138 server_base.cc:1034] running on GCE node
I20260620 23:00:58.230860 22138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:58.231227 22138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:58.232473 22138 hybrid_clock.cc:648] HybridClock initialized: now 1781996458232432 us; error 75 us; skew 500 ppm
Jun 20 23:00:58 dist-test-slave-rdl9 krb5kdc[22125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:58.235489 22138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:00:58.236030 22138 webserver.cc:460] Webserver started at http://127.0.0.1:37887/ using document root <none> and password file <none>
I20260620 23:00:58.236276 22138 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:58.236328 22138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:58.236459 22138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:58.237649 22138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "d881240b3b874136ab8dfdbb6326932b"
format_stamp: "Formatted at 2026-06-20 23:00:58 on dist-test-slave-rdl9"
server_key: "db52816d13716a0faad070af044fd673"
server_key_iv: "ed11dcf162b486259b57e19023e31ef8"
server_key_version: "encryptionkey@0"
I20260620 23:00:58.238004 22138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "d881240b3b874136ab8dfdbb6326932b"
format_stamp: "Formatted at 2026-06-20 23:00:58 on dist-test-slave-rdl9"
server_key: "db52816d13716a0faad070af044fd673"
server_key_iv: "ed11dcf162b486259b57e19023e31ef8"
server_key_version: "encryptionkey@0"
I20260620 23:00:58.239466 22138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:00:58.240310 22153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:58.240548 22138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:58.240643 22138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "d881240b3b874136ab8dfdbb6326932b"
format_stamp: "Formatted at 2026-06-20 23:00:58 on dist-test-slave-rdl9"
server_key: "db52816d13716a0faad070af044fd673"
server_key_iv: "ed11dcf162b486259b57e19023e31ef8"
server_key_version: "encryptionkey@0"
I20260620 23:00:58.240731 22138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:58.244230 22138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:58.246694 22138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:58.246840 22138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:58.251989 22138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42017
I20260620 23:00:58.252032 22205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42017 every 8 connection(s)
I20260620 23:00:58.252382 22138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:58.253726 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:58.256815 22206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b: Bootstrap starting.
I20260620 23:00:58.257437 22206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:58.257694 22206 log.cc:824] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:58.258459 22206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b: No bootstrap required, opened a new log
I20260620 23:00:58.260143 22206 raft_consensus.cc:348] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d881240b3b874136ab8dfdbb6326932b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42017 } }
I20260620 23:00:58.260396 22206 raft_consensus.cc:374] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:58.260502 22206 raft_consensus.cc:729] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d881240b3b874136ab8dfdbb6326932b, State: Initialized, Role: FOLLOWER
I20260620 23:00:58.260690 22206 consensus_queue.cc:260] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d881240b3b874136ab8dfdbb6326932b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42017 } }
I20260620 23:00:58.260800 22206 raft_consensus.cc:388] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:58.260826 22206 raft_consensus.cc:482] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:58.260859 22206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:58.261178 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22138
I20260620 23:00:58.261432 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:58.261648 1688 external_mini_cluster.cc:1428] Setting key f178ab47395b402580fa5a852e65fc59
I20260620 23:00:58.262128 22206 raft_consensus.cc:504] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d881240b3b874136ab8dfdbb6326932b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42017 } }
I20260620 23:00:58.262301 22206 leader_election.cc:302] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d881240b3b874136ab8dfdbb6326932b; no voters:
I20260620 23:00:58.262562 22206 leader_election.cc:288] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:58.262825 22210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:58.262909 22206 sys_catalog.cc:564] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:58.262974 22210 raft_consensus.cc:686] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [term 1 LEADER]: Becoming Leader. State: Replica: d881240b3b874136ab8dfdbb6326932b, State: Running, Role: LEADER
I20260620 23:00:58.263073 22210 consensus_queue.cc:237] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d881240b3b874136ab8dfdbb6326932b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42017 } }
I20260620 23:00:58.263530 22212 sys_catalog.cc:455] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d881240b3b874136ab8dfdbb6326932b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d881240b3b874136ab8dfdbb6326932b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42017 } } }
I20260620 23:00:58.263530 22213 sys_catalog.cc:455] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d881240b3b874136ab8dfdbb6326932b. Latest consensus state: current_term: 1 leader_uuid: "d881240b3b874136ab8dfdbb6326932b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d881240b3b874136ab8dfdbb6326932b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42017 } } }
I20260620 23:00:58.263646 22212 sys_catalog.cc:458] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:58.263665 22213 sys_catalog.cc:458] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:58.263870 22220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:58.264446 22220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:58.266371 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:58.262455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59401 (local address 127.0.0.1:42017)
0620 23:00:58.264545 (+ 2090us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:58.264556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:58.264599 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:00:58.264734 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:58.264743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:58.264790 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:58.264975 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:58.265778 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:58.265782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:58.265783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:58.265788 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:58.265834 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:58.265857 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:58.265918 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:58.266110 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:00:58.266164 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2027,"thread_start_us":91,"threads_started":1}
I20260620 23:00:58.266672 22220 catalog_manager.cc:1347] Generated new cluster ID: dd24350b445c4320ac59fe58a1ee6f44
I20260620 23:00:58.266734 22220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:58.267386 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:00:58.270102 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:58.268879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38007 (local address 127.0.0.1:42017)
0620 23:00:58.268927 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:58.268929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:58.269028 (+ 99us) server_negotiation.cc:407] Connection header received
0620 23:00:58.269105 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:58.269106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:58.269116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:58.269180 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:58.269820 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:58.269822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:58.269822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:00:58.269827 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:58.269854 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:00:58.269860 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:58.269918 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:58.269981 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 23:00:58.269999 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:00:58.279470 22220 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:58.279733 22220 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:58.282979 22220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d881240b3b874136ab8dfdbb6326932b: Generated new TSK 0
I20260620 23:00:58.283227 22220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:58.288527 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22138
2026-06-20T23:00:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:00:58 dist-test-slave-rdl9 krb5kdc[22237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:00:58 dist-test-slave-rdl9 krb5kdc[22237](info): set up 2 sockets
Jun 20 23:00:58 dist-test-slave-rdl9 krb5kdc[22237](info): commencing operation
krb5kdc: starting...
W20260620 23:00:59.278393 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:00:59 dist-test-slave-rdl9 krb5kdc[22237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:00:59.407892 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:00:59.496521 22250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:59.496780 22250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:59.496815 22250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:59.498208 22250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:59.498288 22250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:00:59.498308 22250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:59.498325 22250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:59.498363 22250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:59.500125 22250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:00:59.500442 22250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:59.500679 22250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:59.505769 22258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:59.505769 22255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:59.505769 22256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:59.506068 22250 server_base.cc:1034] running on GCE node
I20260620 23:00:59.506307 22250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:59.506659 22250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:59.507831 22250 hybrid_clock.cc:648] HybridClock initialized: now 1781996459507813 us; error 30 us; skew 500 ppm
Jun 20 23:00:59 dist-test-slave-rdl9 krb5kdc[22237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:00:59.510596 22250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:00:59.511114 22250 webserver.cc:460] Webserver started at http://127.0.0.1:42179/ using document root <none> and password file <none>
I20260620 23:00:59.511327 22250 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:59.511375 22250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:59.511502 22250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:59.512595 22250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "890d74b5951347458d0376103715f6e4"
format_stamp: "Formatted at 2026-06-20 23:00:59 on dist-test-slave-rdl9"
server_key: "aa939e4a3de83dd15dd3339e0da079b2"
server_key_iv: "af8f30e80ae993ca3618745596a7d2f1"
server_key_version: "encryptionkey@0"
I20260620 23:00:59.512949 22250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "890d74b5951347458d0376103715f6e4"
format_stamp: "Formatted at 2026-06-20 23:00:59 on dist-test-slave-rdl9"
server_key: "aa939e4a3de83dd15dd3339e0da079b2"
server_key_iv: "af8f30e80ae993ca3618745596a7d2f1"
server_key_version: "encryptionkey@0"
I20260620 23:00:59.514614 22250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:00:59.515610 22265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:59.516005 22250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:00:59.516108 22250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "890d74b5951347458d0376103715f6e4"
format_stamp: "Formatted at 2026-06-20 23:00:59 on dist-test-slave-rdl9"
server_key: "aa939e4a3de83dd15dd3339e0da079b2"
server_key_iv: "af8f30e80ae993ca3618745596a7d2f1"
server_key_version: "encryptionkey@0"
I20260620 23:00:59.516186 22250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:59.526185 22250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:59.528900 22250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:59.529098 22250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:59.533473 22250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45521
I20260620 23:00:59.533521 22317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45521 every 8 connection(s)
I20260620 23:00:59.533869 22250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:00:59.534560 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22250
I20260620 23:00:59.534683 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:00:59.534858 1688 external_mini_cluster.cc:1428] Setting key 80b9b46017c217fb77f919b4278a5398
I20260620 23:00:59.535089 22318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:59.538652 22318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4: Bootstrap starting.
I20260620 23:00:59.539291 22318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:59.539551 22318 log.cc:824] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:59.540215 22318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4: No bootstrap required, opened a new log
Jun 20 23:00:59 dist-test-slave-rdl9 krb5kdc[22237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:00:59.542498 22318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d74b5951347458d0376103715f6e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45521 } }
I20260620 23:00:59.542667 22318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:59.542691 22318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890d74b5951347458d0376103715f6e4, State: Initialized, Role: FOLLOWER
I20260620 23:00:59.542829 22318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d74b5951347458d0376103715f6e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45521 } }
I20260620 23:00:59.542900 22318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:59.542923 22318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:59.542972 22318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:59.543742 22318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d74b5951347458d0376103715f6e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45521 } }
I20260620 23:00:59.543877 22318 leader_election.cc:302] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890d74b5951347458d0376103715f6e4; no voters:
I20260620 23:00:59.544075 22318 leader_election.cc:288] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:59.544243 22323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:59.544370 22318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:59.544409 22323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [term 1 LEADER]: Becoming Leader. State: Replica: 890d74b5951347458d0376103715f6e4, State: Running, Role: LEADER
I20260620 23:00:59.544507 22323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d74b5951347458d0376103715f6e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45521 } }
I20260620 23:00:59.544934 22325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890d74b5951347458d0376103715f6e4. Latest consensus state: current_term: 1 leader_uuid: "890d74b5951347458d0376103715f6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d74b5951347458d0376103715f6e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45521 } } }
I20260620 23:00:59.545054 22325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:59.545254 22332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:59.545397 22324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890d74b5951347458d0376103715f6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d74b5951347458d0376103715f6e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45521 } } }
I20260620 23:00:59.545483 22324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:59.545917 22332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:59.547538 22332 catalog_manager.cc:1347] Generated new cluster ID: 73563e6d065d438bb8571571c4e3f96a
I20260620 23:00:59.547591 22332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:59.548625 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:59.535397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57949 (local address 127.0.0.1:45521)
0620 23:00:59.535711 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:59.535724 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:59.535760 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:00:59.536435 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:59.536445 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:59.536512 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:59.536682 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:59.537269 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:59.538634 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:59.539424 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:59.539869 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:59.539964 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:59.543122 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:59.543126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:59.543127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:59.543142 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:59.545977 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:59.546516 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:59.546519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:59.546520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:59.546641 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:59.546866 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:59.546867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:59.546867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:59.547182 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:59.547337 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:59.548146 (+ 809us) server_negotiation.cc:299] Negotiation successful
0620 23:00:59.548359 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260620 23:00:59.549865 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:00:59.560019 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:00:59.551531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49047 (local address 127.0.0.1:45521)
0620 23:00:59.551581 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:00:59.551583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:00:59.551728 (+ 145us) server_negotiation.cc:407] Connection header received
0620 23:00:59.551797 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:00:59.551798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:00:59.551809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:00:59.551859 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:00:59.552555 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:59.553616 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:59.554704 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:00:59.555015 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:00:59.555084 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:00:59.557034 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:00:59.557036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:00:59.557036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:00:59.557044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:00:59.558718 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:59.559213 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:59.559214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:59.559214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:59.559274 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:00:59.559460 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:00:59.559461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:00:59.559461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:00:59.559590 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:00:59.559683 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:00:59.559823 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 23:00:59.559915 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:00:59.574163 22332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:59.575428 22332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:59.590377 22332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890d74b5951347458d0376103715f6e4: Generated new TSK 0
I20260620 23:00:59.590685 22332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:59.600507 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22250
2026-06-20T23:00:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:00:59 dist-test-slave-rdl9 krb5kdc[22349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:00:59 dist-test-slave-rdl9 krb5kdc[22349](info): set up 2 sockets
Jun 20 23:00:59 dist-test-slave-rdl9 krb5kdc[22349](info): commencing operation
krb5kdc: starting...
W20260620 23:01:00.587004 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:00 dist-test-slave-rdl9 krb5kdc[22349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:01:00.718191 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:00.810858 22362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:00.811100 22362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:00.811139 22362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:00.812355 22362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:00.812409 22362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:00.812428 22362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:00.812444 22362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:00.812459 22362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:00.814040 22362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:00.814312 22362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:00.814618 22362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:00.819593 22368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:00.819602 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:00.819833 22362 server_base.cc:1034] running on GCE node
W20260620 23:01:00.819593 22367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:00.820284 22362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:00.820677 22362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:00.821903 22362 hybrid_clock.cc:648] HybridClock initialized: now 1781996460821880 us; error 41 us; skew 500 ppm
Jun 20 23:01:00 dist-test-slave-rdl9 krb5kdc[22349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:00.828248 22362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:01:00.828828 22362 webserver.cc:460] Webserver started at http://127.0.0.1:37385/ using document root <none> and password file <none>
I20260620 23:01:00.829066 22362 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:00.829118 22362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:00.829250 22362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:00.830425 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b"
format_stamp: "Formatted at 2026-06-20 23:01:00 on dist-test-slave-rdl9"
server_key: "c876b14a70ddb6a598d35e619586e977"
server_key_iv: "3597974a76a8e96aa3d9d9ee858bb862"
server_key_version: "encryptionkey@0"
I20260620 23:01:00.830791 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b"
format_stamp: "Formatted at 2026-06-20 23:01:00 on dist-test-slave-rdl9"
server_key: "c876b14a70ddb6a598d35e619586e977"
server_key_iv: "3597974a76a8e96aa3d9d9ee858bb862"
server_key_version: "encryptionkey@0"
I20260620 23:01:00.832206 22362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:01:00.833195 22377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:00.833487 22362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:00.833712 22362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b"
format_stamp: "Formatted at 2026-06-20 23:01:00 on dist-test-slave-rdl9"
server_key: "c876b14a70ddb6a598d35e619586e977"
server_key_iv: "3597974a76a8e96aa3d9d9ee858bb862"
server_key_version: "encryptionkey@0"
I20260620 23:01:00.833815 22362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:00.845731 22362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:00.848096 22362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:00.848255 22362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:00.852571 22362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43717
I20260620 23:01:00.852634 22429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43717 every 8 connection(s)
I20260620 23:01:00.853003 22362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:00.854089 22430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:00.855276 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22362
I20260620 23:01:00.855396 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:00.855623 1688 external_mini_cluster.cc:1428] Setting key e25c9b605af79c8fb2f9744bbfacc35d
I20260620 23:01:00.857252 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b: Bootstrap starting.
I20260620 23:01:00.858099 22430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:00.858465 22430 log.cc:824] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:00.859297 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b: No bootstrap required, opened a new log
I20260620 23:01:00.861428 22430 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43717 } }
I20260620 23:01:00.861622 22430 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:00.861989 22430 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb85d6d7ce4847589a2f2b25a5e3ec1b, State: Initialized, Role: FOLLOWER
I20260620 23:01:00.862304 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:00.856325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57041 (local address 127.0.0.1:43717)
0620 23:01:00.856508 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:00.856516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:00.856546 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:01:00.856992 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:00.856999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:00.857060 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:00.857436 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:00.857776 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:00.858905 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:00.859816 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:00.860285 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:00.860403 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:00.861139 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:00.861143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:00.861144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:00.861318 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:00.861354 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:00.861384 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:00.861442 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:00.861530 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:01:00.861613 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260620 23:01:00.862236 22430 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43717 } }
I20260620 23:01:00.862468 22430 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:00.862527 22430 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:00.862596 22430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:00.863754 22430 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43717 } }
I20260620 23:01:00.863999 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:00.864014 22430 leader_election.cc:302] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb85d6d7ce4847589a2f2b25a5e3ec1b; no voters:
I20260620 23:01:00.864202 22430 leader_election.cc:288] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:00.864256 22435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:00.864387 22435 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [term 1 LEADER]: Becoming Leader. State: Replica: eb85d6d7ce4847589a2f2b25a5e3ec1b, State: Running, Role: LEADER
I20260620 23:01:00.864507 22430 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:00.864495 22435 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43717 } }
I20260620 23:01:00.864897 22435 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb85d6d7ce4847589a2f2b25a5e3ec1b. Latest consensus state: current_term: 1 leader_uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43717 } } }
I20260620 23:01:00.864995 22435 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:00.864904 22437 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb85d6d7ce4847589a2f2b25a5e3ec1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43717 } } }
I20260620 23:01:00.865175 22437 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:00.865229 22441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:00.865869 22441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:00.868108 22441 catalog_manager.cc:1347] Generated new cluster ID: a4dcaf59bc2940bf834d01d27b2795de
I20260620 23:01:00.868188 22441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:00.871973 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:00.866942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33489 (local address 127.0.0.1:43717)
0620 23:01:00.866989 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:00.866992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:00.867372 (+ 380us) server_negotiation.cc:407] Connection header received
0620 23:01:00.867523 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:00.867524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:00.867539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:00.867855 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:00.868087 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:00.868923 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:00.870180 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:00.870723 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:00.870839 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:00.871699 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:00.871700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:00.871700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:00.871707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:00.871721 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:00.871725 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:00.871783 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:00.871845 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 23:01:00.871876 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:00.890508 22441 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:00.891397 22441 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:00.897933 22441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb85d6d7ce4847589a2f2b25a5e3ec1b: Generated new TSK 0
I20260620 23:01:00.898232 22441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:00.903173 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22362
2026-06-20T23:01:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:00 dist-test-slave-rdl9 krb5kdc[22461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:00 dist-test-slave-rdl9 krb5kdc[22461](info): set up 2 sockets
Jun 20 23:01:00 dist-test-slave-rdl9 krb5kdc[22461](info): commencing operation
krb5kdc: starting...
W20260620 23:01:01.890834 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:01 dist-test-slave-rdl9 krb5kdc[22461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:01:02.018415 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:02.112478 22474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:02.112721 22474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:02.112747 22474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:02.114063 22474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:02.114133 22474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:02.114147 22474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:02.114159 22474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:02.114171 22474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:02.115792 22474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:02.116035 22474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:02.116258 22474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:02.120779 22482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:02.120836 22480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:02.120806 22479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:02.121069 22474 server_base.cc:1034] running on GCE node
I20260620 23:01:02.121305 22474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:02.121569 22474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:02.122720 22474 hybrid_clock.cc:648] HybridClock initialized: now 1781996462122700 us; error 30 us; skew 500 ppm
Jun 20 23:01:02 dist-test-slave-rdl9 krb5kdc[22461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:02.125089 22474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:01:02.125505 22474 webserver.cc:460] Webserver started at http://127.0.0.1:33297/ using document root <none> and password file <none>
I20260620 23:01:02.125702 22474 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:02.125748 22474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:02.125871 22474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:02.127096 22474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "7c064afe80644347ba013a8ac3b430e0"
format_stamp: "Formatted at 2026-06-20 23:01:02 on dist-test-slave-rdl9"
server_key: "0b3095b12225a1b8448d968092ab8aae"
server_key_iv: "7286d5eb227d06a6ffcaf349173a86da"
server_key_version: "encryptionkey@0"
I20260620 23:01:02.127456 22474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "7c064afe80644347ba013a8ac3b430e0"
format_stamp: "Formatted at 2026-06-20 23:01:02 on dist-test-slave-rdl9"
server_key: "0b3095b12225a1b8448d968092ab8aae"
server_key_iv: "7286d5eb227d06a6ffcaf349173a86da"
server_key_version: "encryptionkey@0"
I20260620 23:01:02.129070 22474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:02.130098 22489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:02.130424 22474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:02.130518 22474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "7c064afe80644347ba013a8ac3b430e0"
format_stamp: "Formatted at 2026-06-20 23:01:02 on dist-test-slave-rdl9"
server_key: "0b3095b12225a1b8448d968092ab8aae"
server_key_iv: "7286d5eb227d06a6ffcaf349173a86da"
server_key_version: "encryptionkey@0"
I20260620 23:01:02.130592 22474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:02.133984 22474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:02.136559 22474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:02.136821 22474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:02.141911 22474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37327
I20260620 23:01:02.141942 22541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37327 every 8 connection(s)
I20260620 23:01:02.142422 22474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:02.143580 22542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:02.143899 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22474
I20260620 23:01:02.144025 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:02.144213 1688 external_mini_cluster.cc:1428] Setting key 211abf9b080f8b926ea7bcaab881a084
I20260620 23:01:02.147359 22542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0: Bootstrap starting.
I20260620 23:01:02.147382 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:02.144851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48057 (local address 127.0.0.1:37327)
0620 23:01:02.145077 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:02.145088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:02.145127 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:01:02.145666 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:02.145675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:02.145711 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:02.145877 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:02.146610 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:02.146618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:02.146618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:02.146627 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:02.146715 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:02.146736 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:02.146862 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:02.146977 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 23:01:02.147037 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260620 23:01:02.148147 22542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:02.148700 22542 log.cc:824] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:02.148752 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:02.149677 22542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0: No bootstrap required, opened a new log
I20260620 23:01:02.151428 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:02.150269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49087 (local address 127.0.0.1:37327)
0620 23:01:02.150322 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:02.150324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:02.150503 (+ 179us) server_negotiation.cc:407] Connection header received
0620 23:01:02.150573 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:02.150574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:02.150580 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:02.150627 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:02.151210 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:02.151211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:02.151212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:02.151216 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:02.151233 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:02.151237 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:02.151285 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:02.151333 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 23:01:02.151349 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:01:02.152293 22542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c064afe80644347ba013a8ac3b430e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37327 } }
I20260620 23:01:02.152505 22542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:02.152530 22542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c064afe80644347ba013a8ac3b430e0, State: Initialized, Role: FOLLOWER
I20260620 23:01:02.152673 22542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c064afe80644347ba013a8ac3b430e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37327 } }
I20260620 23:01:02.152769 22542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:02.152798 22542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:02.152853 22542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:02.153573 22542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c064afe80644347ba013a8ac3b430e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37327 } }
I20260620 23:01:02.153697 22542 leader_election.cc:302] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c064afe80644347ba013a8ac3b430e0; no voters:
I20260620 23:01:02.153909 22542 leader_election.cc:288] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:02.154155 22552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:02.154193 22542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:02.154475 22552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c064afe80644347ba013a8ac3b430e0, State: Running, Role: LEADER
I20260620 23:01:02.154739 22552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c064afe80644347ba013a8ac3b430e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37327 } }
I20260620 23:01:02.155483 22553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c064afe80644347ba013a8ac3b430e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c064afe80644347ba013a8ac3b430e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37327 } } }
I20260620 23:01:02.155596 22553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:02.155836 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c064afe80644347ba013a8ac3b430e0. Latest consensus state: current_term: 1 leader_uuid: "7c064afe80644347ba013a8ac3b430e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c064afe80644347ba013a8ac3b430e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37327 } } }
I20260620 23:01:02.155872 22566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:02.156219 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:02.156874 22566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:02.158782 22566 catalog_manager.cc:1347] Generated new cluster ID: c4b5340234cd4a3fa59003305aa9ae59
I20260620 23:01:02.158875 22566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:02.165576 22566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:02.165771 22566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:02.176249 22566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c064afe80644347ba013a8ac3b430e0: Generated new TSK 0
I20260620 23:01:02.176493 22566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:02.181495 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22474
2026-06-20T23:01:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:02 dist-test-slave-rdl9 krb5kdc[22573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:02 dist-test-slave-rdl9 krb5kdc[22573](info): set up 2 sockets
Jun 20 23:01:02 dist-test-slave-rdl9 krb5kdc[22573](info): commencing operation
krb5kdc: starting...
W20260620 23:01:03.170853 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:03 dist-test-slave-rdl9 krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:01:03.302680 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:03.391322 22586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:03.391804 22586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:03.391923 22586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:03.394518 22586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:03.394657 22586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:03.394685 22586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:03.394713 22586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:03.394739 22586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:03.397316 22586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:03.397653 22586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:03.397926 22586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:03.403748 22591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:03.403841 22586 server_base.cc:1034] running on GCE node
W20260620 23:01:03.403740 22592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:03.403749 22594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:03.404219 22586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:03.404662 22586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:03.405834 22586 hybrid_clock.cc:648] HybridClock initialized: now 1781996463405794 us; error 56 us; skew 500 ppm
Jun 20 23:01:03 dist-test-slave-rdl9 krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:03.408874 22586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:01:03.409348 22586 webserver.cc:460] Webserver started at http://127.0.0.1:33517/ using document root <none> and password file <none>
I20260620 23:01:03.409581 22586 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:03.409629 22586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:03.409736 22586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:03.411202 22586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "7e1fc2272e284ace96f37ff909f93152"
format_stamp: "Formatted at 2026-06-20 23:01:03 on dist-test-slave-rdl9"
server_key: "2f683109e50fdab2b421600fc1d7c0f3"
server_key_iv: "9274dd149f947375f25657f43eb3856b"
server_key_version: "encryptionkey@0"
I20260620 23:01:03.411589 22586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "7e1fc2272e284ace96f37ff909f93152"
format_stamp: "Formatted at 2026-06-20 23:01:03 on dist-test-slave-rdl9"
server_key: "2f683109e50fdab2b421600fc1d7c0f3"
server_key_iv: "9274dd149f947375f25657f43eb3856b"
server_key_version: "encryptionkey@0"
I20260620 23:01:03.413187 22586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:03.414258 22601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:03.414477 22586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:03.414589 22586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "7e1fc2272e284ace96f37ff909f93152"
format_stamp: "Formatted at 2026-06-20 23:01:03 on dist-test-slave-rdl9"
server_key: "2f683109e50fdab2b421600fc1d7c0f3"
server_key_iv: "9274dd149f947375f25657f43eb3856b"
server_key_version: "encryptionkey@0"
I20260620 23:01:03.414670 22586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:03.452562 22586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:03.454921 22586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:03.455067 22586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:03.459794 22586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43691
I20260620 23:01:03.459944 22653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43691 every 8 connection(s)
I20260620 23:01:03.460219 22586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:03.461257 22654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:03.463709 22654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152: Bootstrap starting.
I20260620 23:01:03.464339 22654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:03.464599 22654 log.cc:824] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:03.465313 22654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152: No bootstrap required, opened a new log
I20260620 23:01:03.467003 22654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1fc2272e284ace96f37ff909f93152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260620 23:01:03.467157 22654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:03.467180 22654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1fc2272e284ace96f37ff909f93152, State: Initialized, Role: FOLLOWER
I20260620 23:01:03.467319 22654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1fc2272e284ace96f37ff909f93152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260620 23:01:03.467391 22654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:03.467412 22654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:03.467451 22654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:03.468076 22654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1fc2272e284ace96f37ff909f93152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260620 23:01:03.468168 22654 leader_election.cc:302] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e1fc2272e284ace96f37ff909f93152; no voters:
I20260620 23:01:03.468348 22654 leader_election.cc:288] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:03.468441 22657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:03.468636 22657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [term 1 LEADER]: Becoming Leader. State: Replica: 7e1fc2272e284ace96f37ff909f93152, State: Running, Role: LEADER
I20260620 23:01:03.468713 22654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:03.469009 22657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1fc2272e284ace96f37ff909f93152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260620 23:01:03.469894 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22586
I20260620 23:01:03.470021 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:03.469947 22659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e1fc2272e284ace96f37ff909f93152. Latest consensus state: current_term: 1 leader_uuid: "7e1fc2272e284ace96f37ff909f93152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1fc2272e284ace96f37ff909f93152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } } }
I20260620 23:01:03.469959 22658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e1fc2272e284ace96f37ff909f93152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e1fc2272e284ace96f37ff909f93152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } } }
I20260620 23:01:03.470139 22659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:03.470206 1688 external_mini_cluster.cc:1428] Setting key 05421b23cf25f0989e0b4a25ebfdead9
I20260620 23:01:03.470147 22658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152 [sys.catalog]: This master's current role is: LEADER
W20260620 23:01:03.471513 22671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:03.471731 22671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:03.471791 22672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:03.471971 22672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:03.473567 22672 catalog_manager.cc:1347] Generated new cluster ID: d72d43993d354209a93f123428eed75d
I20260620 23:01:03.473626 22672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:01:03 dist-test-slave-rdl9 krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:01:03.481796 22672 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:03.482584 22672 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:03.483064 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:03.470955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:33627 (local address 127.0.0.1:43691)
0620 23:01:03.471144 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:03.471155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:03.471190 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:01:03.471327 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:03.471334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:03.471397 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:03.471546 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:03.472309 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:03.473458 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:03.474580 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:03.475209 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:03.475305 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:03.478170 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:03.478175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:03.478176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:03.478190 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:03.480610 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:03.481326 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:03.481330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:03.481330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:03.481463 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:03.481857 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:03.481858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:03.481858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:03.482164 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:03.482380 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:03.482542 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:01:03.482664 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260620 23:01:03.484149 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:03.494210 22672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e1fc2272e284ace96f37ff909f93152: Generated new TSK 0
I20260620 23:01:03.494659 22672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:03.495846 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:03.485631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:45683 (local address 127.0.0.1:43691)
0620 23:01:03.485667 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:03.485668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:03.485894 (+ 226us) server_negotiation.cc:407] Connection header received
0620 23:01:03.485948 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:03.485949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:03.485961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:03.486032 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:03.486681 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:03.488664 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:03.489791 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:03.490122 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:03.490202 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:03.492161 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:03.492165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:03.492165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:03.492181 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:03.494120 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:03.494840 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:03.494842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:03.494842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:03.494929 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:03.495187 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:03.495188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:03.495188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:03.495350 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:03.495470 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:03.495650 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:01:03.495731 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:01:03.498663 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22586
W20260620 23:01:03.503852 22571 connection.cc:535] client connection to 127.0.0.1:43691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43691): Connection reset by peer (error 104)
2026-06-20T23:01:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:03 dist-test-slave-rdl9 krb5kdc[22685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:03 dist-test-slave-rdl9 krb5kdc[22685](info): set up 2 sockets
Jun 20 23:01:03 dist-test-slave-rdl9 krb5kdc[22685](info): commencing operation
krb5kdc: starting...
W20260620 23:01:04.495112 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:04 dist-test-slave-rdl9 krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:01:04.617717 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:04.715212 22698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:04.715557 22698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:04.715606 22698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:04.717815 22698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:04.717883 22698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:04.717904 22698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:04.717921 22698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:04.717938 22698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:04.720481 22698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:04.720889 22698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:04.721146 22698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:04.726599 22704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:04.726611 22706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:04.726604 22703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:04.726636 22698 server_base.cc:1034] running on GCE node
I20260620 23:01:04.727146 22698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:04.727465 22698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:04.728662 22698 hybrid_clock.cc:648] HybridClock initialized: now 1781996464728600 us; error 72 us; skew 500 ppm
Jun 20 23:01:04 dist-test-slave-rdl9 krb5kdc[22685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:04.731539 22698 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:01:04.732023 22698 webserver.cc:460] Webserver started at http://127.0.0.1:43513/ using document root <none> and password file <none>
I20260620 23:01:04.732263 22698 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:04.732314 22698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:04.732460 22698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:04.733613 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "ee6fe90e790d4fb591098b6beead4940"
format_stamp: "Formatted at 2026-06-20 23:01:04 on dist-test-slave-rdl9"
server_key: "77ae6eac436941b548aeb85f978e945f"
server_key_iv: "2de420ff7ba95ce6cf7fd5ede5507d80"
server_key_version: "encryptionkey@0"
I20260620 23:01:04.733960 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "ee6fe90e790d4fb591098b6beead4940"
format_stamp: "Formatted at 2026-06-20 23:01:04 on dist-test-slave-rdl9"
server_key: "77ae6eac436941b548aeb85f978e945f"
server_key_iv: "2de420ff7ba95ce6cf7fd5ede5507d80"
server_key_version: "encryptionkey@0"
I20260620 23:01:04.735723 22698 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:01:04.736985 22713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:04.737272 22698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:04.737409 22698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "ee6fe90e790d4fb591098b6beead4940"
format_stamp: "Formatted at 2026-06-20 23:01:04 on dist-test-slave-rdl9"
server_key: "77ae6eac436941b548aeb85f978e945f"
server_key_iv: "2de420ff7ba95ce6cf7fd5ede5507d80"
server_key_version: "encryptionkey@0"
I20260620 23:01:04.737499 22698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:04.754163 22698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:04.757027 22698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:04.757228 22698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:04.761722 22698 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35877
I20260620 23:01:04.761767 22765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35877 every 8 connection(s)
I20260620 23:01:04.762116 22698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:04.763139 22766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:04.763664 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22698
I20260620 23:01:04.763778 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:04.763948 1688 external_mini_cluster.cc:1428] Setting key 5d84448669436b9f62849275bda4be75
I20260620 23:01:04.765758 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940: Bootstrap starting.
I20260620 23:01:04.766326 22766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:04.766650 22766 log.cc:824] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:04.767274 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940: No bootstrap required, opened a new log
I20260620 23:01:04.769310 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:04.764479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:36881 (local address 127.0.0.1:35877)
0620 23:01:04.764663 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:04.764671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:04.764697 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:01:04.765121 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:04.765127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:04.765177 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:04.765296 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:04.765503 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:04.766477 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:04.767245 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:04.767698 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:04.767793 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:04.768722 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:04.768726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:04.768727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:04.768902 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:04.768934 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:04.768946 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:04.769018 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:04.769111 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 23:01:04.769139 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260620 23:01:04.769747 22766 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6fe90e790d4fb591098b6beead4940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260620 23:01:04.770054 22766 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:04.770097 22766 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6fe90e790d4fb591098b6beead4940, State: Initialized, Role: FOLLOWER
I20260620 23:01:04.770448 22766 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6fe90e790d4fb591098b6beead4940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260620 23:01:04.770618 22766 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:04.770676 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:04.770699 22766 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:04.770875 22766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:04.771808 22766 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6fe90e790d4fb591098b6beead4940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260620 23:01:04.772002 22766 leader_election.cc:302] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee6fe90e790d4fb591098b6beead4940; no voters:
I20260620 23:01:04.772256 22766 leader_election.cc:288] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:04.772330 22775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:04.772524 22775 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [term 1 LEADER]: Becoming Leader. State: Replica: ee6fe90e790d4fb591098b6beead4940, State: Running, Role: LEADER
I20260620 23:01:04.772696 22766 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:04.772683 22775 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6fe90e790d4fb591098b6beead4940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260620 23:01:04.773232 22778 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee6fe90e790d4fb591098b6beead4940. Latest consensus state: current_term: 1 leader_uuid: "ee6fe90e790d4fb591098b6beead4940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6fe90e790d4fb591098b6beead4940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } } }
I20260620 23:01:04.773262 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee6fe90e790d4fb591098b6beead4940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6fe90e790d4fb591098b6beead4940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } } }
I20260620 23:01:04.773454 22778 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:04.773476 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:04.773900 22785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:04.774427 22785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:04.776353 22785 catalog_manager.cc:1347] Generated new cluster ID: 8d7ca8d926a849268ecfcda41acc4df9
I20260620 23:01:04.776409 22785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:04.778472 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:04.772528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:43017 (local address 127.0.0.1:35877)
0620 23:01:04.772599 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:04.772601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:04.774579 (+ 1978us) server_negotiation.cc:407] Connection header received
0620 23:01:04.774705 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:04.774707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:04.774719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:04.774771 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:04.774969 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:04.775531 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:04.776725 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:04.777067 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:04.777137 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:04.778094 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:04.778096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:04.778096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:04.778102 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:04.778117 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:04.778124 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:04.778175 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:04.778377 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:01:04.778391 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:01:04.785643 22785 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:04.786787 22785 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:04.792616 22785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee6fe90e790d4fb591098b6beead4940: Generated new TSK 0
I20260620 23:01:04.792873 22785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:04.797508 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22698
W20260620 23:01:04.802007 22683 connection.cc:535] client connection to 127.0.0.1:35877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35877): Connection reset by peer (error 104)
2026-06-20T23:01:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:04 dist-test-slave-rdl9 krb5kdc[22797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:04 dist-test-slave-rdl9 krb5kdc[22797](info): set up 2 sockets
Jun 20 23:01:04 dist-test-slave-rdl9 krb5kdc[22797](info): commencing operation
krb5kdc: starting...
W20260620 23:01:05.770539 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:05 dist-test-slave-rdl9 krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:01:05.898468 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:05.990495 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:05.990948 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:05.990988 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:05.992671 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:05.992764 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:05.992784 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:05.992797 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:05.992815 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:05.994966 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:05.995330 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:05.995623 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:06.001588 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:06.001588 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:06.001588 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:06.001670 22810 server_base.cc:1034] running on GCE node
I20260620 23:01:06.002210 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:06.002607 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:06.003798 22810 hybrid_clock.cc:648] HybridClock initialized: now 1781996466003783 us; error 38 us; skew 500 ppm
Jun 20 23:01:06 dist-test-slave-rdl9 krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:06.011319 22810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:01:06.012058 22810 webserver.cc:460] Webserver started at http://127.0.0.1:35903/ using document root <none> and password file <none>
I20260620 23:01:06.012346 22810 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:06.012404 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:06.012555 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:06.013851 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "aa3f449a76e043a5b637c31b4081e535"
format_stamp: "Formatted at 2026-06-20 23:01:06 on dist-test-slave-rdl9"
server_key: "b0f0911d19b066755c5e745647ec7ca6"
server_key_iv: "24a037aa30f4c5f24eea0f43ff2eee12"
server_key_version: "encryptionkey@0"
I20260620 23:01:06.014249 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "aa3f449a76e043a5b637c31b4081e535"
format_stamp: "Formatted at 2026-06-20 23:01:06 on dist-test-slave-rdl9"
server_key: "b0f0911d19b066755c5e745647ec7ca6"
server_key_iv: "24a037aa30f4c5f24eea0f43ff2eee12"
server_key_version: "encryptionkey@0"
I20260620 23:01:06.016016 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 23:01:06.017231 22825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:06.017530 22810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:06.017661 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "aa3f449a76e043a5b637c31b4081e535"
format_stamp: "Formatted at 2026-06-20 23:01:06 on dist-test-slave-rdl9"
server_key: "b0f0911d19b066755c5e745647ec7ca6"
server_key_iv: "24a037aa30f4c5f24eea0f43ff2eee12"
server_key_version: "encryptionkey@0"
I20260620 23:01:06.017753 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:06.021406 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:06.024096 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:06.024259 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:06.028524 22810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40415
I20260620 23:01:06.028581 22877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40415 every 8 connection(s)
I20260620 23:01:06.029069 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:06.030191 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:06.032819 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535: Bootstrap starting.
I20260620 23:01:06.033483 22878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:06.033761 22878 log.cc:824] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:06.034236 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22810
I20260620 23:01:06.034422 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:06.034802 1688 external_mini_cluster.cc:1428] Setting key 9adabb37339a4c5f76745e7c6dc6568c
I20260620 23:01:06.034904 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535: No bootstrap required, opened a new log
I20260620 23:01:06.037631 22878 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3f449a76e043a5b637c31b4081e535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40415 } }
I20260620 23:01:06.037835 22878 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:06.037880 22878 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3f449a76e043a5b637c31b4081e535, State: Initialized, Role: FOLLOWER
I20260620 23:01:06.038049 22878 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3f449a76e043a5b637c31b4081e535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40415 } }
I20260620 23:01:06.038154 22878 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:06.038192 22878 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:06.038259 22878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:06.038556 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:06.035506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:41569 (local address 127.0.0.1:40415)
0620 23:01:06.036024 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:06.036036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:06.036079 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:01:06.036683 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:06.036694 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:06.036731 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:06.037123 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:06.037753 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:06.037757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:06.037758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:06.037763 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:06.037828 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:06.037844 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:06.037932 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:06.038187 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 23:01:06.038266 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":338,"threads_started":1}
I20260620 23:01:06.039386 22878 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3f449a76e043a5b637c31b4081e535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40415 } }
I20260620 23:01:06.039492 22878 leader_election.cc:302] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa3f449a76e043a5b637c31b4081e535; no voters:
I20260620 23:01:06.039678 22878 leader_election.cc:288] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:06.039722 22883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:06.039809 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:06.039877 22883 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [term 1 LEADER]: Becoming Leader. State: Replica: aa3f449a76e043a5b637c31b4081e535, State: Running, Role: LEADER
I20260620 23:01:06.039956 22883 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3f449a76e043a5b637c31b4081e535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40415 } }
I20260620 23:01:06.039983 22878 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:06.040298 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa3f449a76e043a5b637c31b4081e535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3f449a76e043a5b637c31b4081e535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40415 } } }
I20260620 23:01:06.040378 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:06.040531 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa3f449a76e043a5b637c31b4081e535. Latest consensus state: current_term: 1 leader_uuid: "aa3f449a76e043a5b637c31b4081e535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3f449a76e043a5b637c31b4081e535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40415 } } }
I20260620 23:01:06.040581 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:06.041064 22889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:06.041759 22889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:06.042725 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:06.041501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:40577 (local address 127.0.0.1:40415)
0620 23:01:06.041545 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:06.041547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:06.041557 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:06.041585 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:06.041585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:06.041591 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:06.041693 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:06.042483 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:06.042485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:06.042485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:06.042490 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:06.042507 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:06.042513 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:06.042541 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:06.042640 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 23:01:06.042656 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:01:06.044296 22889 catalog_manager.cc:1347] Generated new cluster ID: b0519ea68cc44816b994f565cd8f2568
I20260620 23:01:06.044381 22889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:06.064093 22889 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:06.064332 22889 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:06.070658 22889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa3f449a76e043a5b637c31b4081e535: Generated new TSK 0
I20260620 23:01:06.070868 22889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:06.079598 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22810
2026-06-20T23:01:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:06 dist-test-slave-rdl9 krb5kdc[22909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:06 dist-test-slave-rdl9 krb5kdc[22909](info): set up 2 sockets
Jun 20 23:01:06 dist-test-slave-rdl9 krb5kdc[22909](info): commencing operation
krb5kdc: starting...
W20260620 23:01:07.059058 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:07 dist-test-slave-rdl9 krb5kdc[22909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:01:07.177736 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:07.260941 22922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:07.261178 22922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:07.261212 22922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:07.263070 22922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:07.263150 22922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:07.263173 22922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:07.263192 22922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:07.263207 22922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:07.265038 22922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:07.265341 22922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:07.265573 22922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:07.270748 22930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:07.270802 22927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:07.270877 22928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:07.271092 22922 server_base.cc:1034] running on GCE node
I20260620 23:01:07.271359 22922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:07.271683 22922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:07.272850 22922 hybrid_clock.cc:648] HybridClock initialized: now 1781996467272836 us; error 35 us; skew 500 ppm
Jun 20 23:01:07 dist-test-slave-rdl9 krb5kdc[22909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:07.275930 22922 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:01:07.276604 22922 webserver.cc:460] Webserver started at http://127.0.0.1:42893/ using document root <none> and password file <none>
I20260620 23:01:07.276867 22922 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:07.276925 22922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:07.277053 22922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:07.278180 22922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "d569aef0e0094fdc9b5e2826733ae21a"
format_stamp: "Formatted at 2026-06-20 23:01:07 on dist-test-slave-rdl9"
server_key: "b95e0c4f97cccaa52cee7858f4ca9022"
server_key_iv: "ffef20731085c17f0356d2828afc6f4a"
server_key_version: "encryptionkey@0"
I20260620 23:01:07.278604 22922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "d569aef0e0094fdc9b5e2826733ae21a"
format_stamp: "Formatted at 2026-06-20 23:01:07 on dist-test-slave-rdl9"
server_key: "b95e0c4f97cccaa52cee7858f4ca9022"
server_key_iv: "ffef20731085c17f0356d2828afc6f4a"
server_key_version: "encryptionkey@0"
I20260620 23:01:07.280068 22922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:07.281038 22937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:07.281345 22922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:07.281509 22922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "d569aef0e0094fdc9b5e2826733ae21a"
format_stamp: "Formatted at 2026-06-20 23:01:07 on dist-test-slave-rdl9"
server_key: "b95e0c4f97cccaa52cee7858f4ca9022"
server_key_iv: "ffef20731085c17f0356d2828afc6f4a"
server_key_version: "encryptionkey@0"
I20260620 23:01:07.281625 22922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:07.297044 22922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:07.299672 22922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:07.299834 22922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:07.303748 22922 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35275
I20260620 23:01:07.303830 22989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35275 every 8 connection(s)
I20260620 23:01:07.304153 22922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:07.305301 22990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:07.307755 22990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a: Bootstrap starting.
I20260620 23:01:07.308372 22990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:07.308641 22990 log.cc:824] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:07.309432 22990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a: No bootstrap required, opened a new log
I20260620 23:01:07.311244 22990 raft_consensus.cc:348] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d569aef0e0094fdc9b5e2826733ae21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } }
I20260620 23:01:07.311419 22990 raft_consensus.cc:374] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:07.311440 22990 raft_consensus.cc:729] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d569aef0e0094fdc9b5e2826733ae21a, State: Initialized, Role: FOLLOWER
I20260620 23:01:07.311669 22990 consensus_queue.cc:260] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d569aef0e0094fdc9b5e2826733ae21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } }
I20260620 23:01:07.311787 22990 raft_consensus.cc:388] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:07.311815 22990 raft_consensus.cc:482] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:07.311856 22990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:07.312489 22990 raft_consensus.cc:504] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d569aef0e0094fdc9b5e2826733ae21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } }
I20260620 23:01:07.312613 22990 leader_election.cc:302] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d569aef0e0094fdc9b5e2826733ae21a; no voters:
I20260620 23:01:07.312801 22990 leader_election.cc:288] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:07.312839 22993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:07.312880 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 22922
I20260620 23:01:07.312964 22993 raft_consensus.cc:686] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [term 1 LEADER]: Becoming Leader. State: Replica: d569aef0e0094fdc9b5e2826733ae21a, State: Running, Role: LEADER
I20260620 23:01:07.313026 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:07.313046 22993 consensus_queue.cc:237] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d569aef0e0094fdc9b5e2826733ae21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } }
I20260620 23:01:07.313122 22990 sys_catalog.cc:564] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:07.313287 1688 external_mini_cluster.cc:1428] Setting key 93742665bde6e08f06c45272dee0ba08
I20260620 23:01:07.313705 22995 sys_catalog.cc:455] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d569aef0e0094fdc9b5e2826733ae21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d569aef0e0094fdc9b5e2826733ae21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } } }
I20260620 23:01:07.313846 22995 sys_catalog.cc:458] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:07.314419 22994 sys_catalog.cc:455] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d569aef0e0094fdc9b5e2826733ae21a. Latest consensus state: current_term: 1 leader_uuid: "d569aef0e0094fdc9b5e2826733ae21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d569aef0e0094fdc9b5e2826733ae21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } } }
I20260620 23:01:07.314532 23004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:07.314613 22994 sys_catalog.cc:458] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:07.315339 23004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:07.317379 23004 catalog_manager.cc:1347] Generated new cluster ID: 8dc3e756c6b24777b95eddca534e38a7
I20260620 23:01:07.317458 23004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:01:07 dist-test-slave-rdl9 krb5kdc[22909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:01:07.326122 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:07.314189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:42535 (local address 127.0.0.1:35275)
0620 23:01:07.314904 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:07.314916 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:07.314955 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:01:07.315095 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:07.315104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:07.315165 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:07.315297 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:07.315921 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:07.317106 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:07.318102 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:07.318946 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:07.319091 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:07.321977 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:07.321983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:07.321984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:07.321998 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:07.324177 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:07.324732 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:07.324736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:07.324737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:07.324873 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:07.325106 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:07.325106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:07.325107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:07.325443 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:07.325602 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:07.325750 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:01:07.325867 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":95,"threads_started":1}
I20260620 23:01:07.327406 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:07.329316 23004 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:07.330117 23004 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:07.335215 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:07.329148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:60871 (local address 127.0.0.1:35275)
0620 23:01:07.329193 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:07.329195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:07.329301 (+ 106us) server_negotiation.cc:407] Connection header received
0620 23:01:07.329387 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:07.329388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:07.329398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:07.329447 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:07.329998 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:07.330607 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:07.331276 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:07.331509 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:07.331565 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:07.332776 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:07.332777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:07.332777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:07.332787 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:07.334048 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:07.334451 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:07.334453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:07.334453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:07.334520 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:07.334717 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:07.334718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:07.334718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:07.334834 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:07.334946 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:07.335056 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 23:01:07.335117 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:01:07.337389 23004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d569aef0e0094fdc9b5e2826733ae21a: Generated new TSK 0
I20260620 23:01:07.337682 23004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:07.343665 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 22922
W20260620 23:01:07.348448 22907 connection.cc:535] client connection to 127.0.0.1:35275 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35275): Connection reset by peer (error 104)
2026-06-20T23:01:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:07 dist-test-slave-rdl9 krb5kdc[23021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:07 dist-test-slave-rdl9 krb5kdc[23021](info): set up 2 sockets
Jun 20 23:01:07 dist-test-slave-rdl9 krb5kdc[23021](info): commencing operation
krb5kdc: starting...
W20260620 23:01:08.318713 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:08 dist-test-slave-rdl9 krb5kdc[23021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:01:08.444895 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:08.533154 23034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:08.533443 23034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:08.533480 23034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:08.535040 23034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:08.535120 23034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:08.535140 23034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:08.535157 23034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:08.535172 23034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:08.537065 23034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34123
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:08.537542 23034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:08.537830 23034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:08.542877 23042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:08.542869 23040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:08.542869 23039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:08.543444 23034 server_base.cc:1034] running on GCE node
I20260620 23:01:08.543875 23034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:08.544317 23034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:08.545543 23034 hybrid_clock.cc:648] HybridClock initialized: now 1781996468545496 us; error 62 us; skew 500 ppm
Jun 20 23:01:08 dist-test-slave-rdl9 krb5kdc[23021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:08.548756 23034 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:01:08.549474 23034 webserver.cc:460] Webserver started at http://127.0.0.1:44419/ using document root <none> and password file <none>
I20260620 23:01:08.549795 23034 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:08.549860 23034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:08.550024 23034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:08.551505 23034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "d598cb4d3c1e4bfcb4a86b79754c7378"
format_stamp: "Formatted at 2026-06-20 23:01:08 on dist-test-slave-rdl9"
server_key: "36adeb6c55075987c5757defde175cb0"
server_key_iv: "345daac974a28338cbf0a0f9f997f388"
server_key_version: "encryptionkey@0"
I20260620 23:01:08.551900 23034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "d598cb4d3c1e4bfcb4a86b79754c7378"
format_stamp: "Formatted at 2026-06-20 23:01:08 on dist-test-slave-rdl9"
server_key: "36adeb6c55075987c5757defde175cb0"
server_key_iv: "345daac974a28338cbf0a0f9f997f388"
server_key_version: "encryptionkey@0"
I20260620 23:01:08.553617 23034 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 23:01:08.554874 23049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:08.555243 23034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:08.555371 23034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "d598cb4d3c1e4bfcb4a86b79754c7378"
format_stamp: "Formatted at 2026-06-20 23:01:08 on dist-test-slave-rdl9"
server_key: "36adeb6c55075987c5757defde175cb0"
server_key_iv: "345daac974a28338cbf0a0f9f997f388"
server_key_version: "encryptionkey@0"
I20260620 23:01:08.555457 23034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:08.566995 23034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:08.569239 23034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:08.569396 23034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:08.573860 23034 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34123
I20260620 23:01:08.573896 23101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34123 every 8 connection(s)
I20260620 23:01:08.574285 23034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:08.575311 23102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:08.577937 23102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378: Bootstrap starting.
I20260620 23:01:08.578755 23102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:08.579041 23102 log.cc:824] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:08.579808 23102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378: No bootstrap required, opened a new log
I20260620 23:01:08.580839 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23034
I20260620 23:01:08.580967 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:08.581190 1688 external_mini_cluster.cc:1428] Setting key 1c87c1467f2d73adef5f57c5f43d769a
I20260620 23:01:08.581693 23102 raft_consensus.cc:348] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d598cb4d3c1e4bfcb4a86b79754c7378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } }
I20260620 23:01:08.581908 23102 raft_consensus.cc:374] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:08.581969 23102 raft_consensus.cc:729] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d598cb4d3c1e4bfcb4a86b79754c7378, State: Initialized, Role: FOLLOWER
I20260620 23:01:08.582146 23102 consensus_queue.cc:260] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d598cb4d3c1e4bfcb4a86b79754c7378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } }
I20260620 23:01:08.582261 23102 raft_consensus.cc:388] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:08.582324 23102 raft_consensus.cc:482] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:08.582417 23102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:08.583256 23102 raft_consensus.cc:504] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d598cb4d3c1e4bfcb4a86b79754c7378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } }
I20260620 23:01:08.583380 23102 leader_election.cc:302] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d598cb4d3c1e4bfcb4a86b79754c7378; no voters:
I20260620 23:01:08.583549 23102 leader_election.cc:288] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:08.583608 23107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:08.583753 23107 raft_consensus.cc:686] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [term 1 LEADER]: Becoming Leader. State: Replica: d598cb4d3c1e4bfcb4a86b79754c7378, State: Running, Role: LEADER
I20260620 23:01:08.583845 23102 sys_catalog.cc:564] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:08.583890 23107 consensus_queue.cc:237] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d598cb4d3c1e4bfcb4a86b79754c7378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } }
I20260620 23:01:08.584259 23107 sys_catalog.cc:455] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d598cb4d3c1e4bfcb4a86b79754c7378. Latest consensus state: current_term: 1 leader_uuid: "d598cb4d3c1e4bfcb4a86b79754c7378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d598cb4d3c1e4bfcb4a86b79754c7378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } } }
I20260620 23:01:08.584321 23109 sys_catalog.cc:455] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d598cb4d3c1e4bfcb4a86b79754c7378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d598cb4d3c1e4bfcb4a86b79754c7378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } } }
I20260620 23:01:08.584390 23107 sys_catalog.cc:458] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:08.584396 23109 sys_catalog.cc:458] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:08.584620 23115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:08.585173 23115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:08.587291 23115 catalog_manager.cc:1347] Generated new cluster ID: ef7e06ad45cf42798b18680abfa21f94
I20260620 23:01:08.587415 23115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:08.587677 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:08.581881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:46511 (local address 127.0.0.1:34123)
0620 23:01:08.582104 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:08.582115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:08.582150 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:01:08.582276 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:08.582284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:08.582381 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:08.582539 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:08.582701 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:08.584432 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:08.585594 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:08.586115 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:08.586225 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:08.586957 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:08.586961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:08.586961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:08.587164 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:08.587204 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:08.587219 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:08.587306 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:08.587435 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:01:08.587464 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
I20260620 23:01:08.588809 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:08.594003 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:08.590249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:45127 (local address 127.0.0.1:34123)
0620 23:01:08.590289 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:08.590291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:08.590438 (+ 147us) server_negotiation.cc:407] Connection header received
0620 23:01:08.590501 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:08.590502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:08.590513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:08.590576 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:08.590707 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:08.591473 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:08.592525 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:08.592859 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:08.592937 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:08.593729 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:08.593730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:08.593731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:08.593738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:08.593757 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:08.593763 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:08.593828 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:08.593888 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:01:08.593904 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:01:08.601425 23115 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:08.601955 23115 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:08.609011 23115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d598cb4d3c1e4bfcb4a86b79754c7378: Generated new TSK 0
I20260620 23:01:08.609323 23115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:08.614488 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23034
W20260620 23:01:08.620543 23019 connection.cc:535] client connection to 127.0.0.1:34123 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34123): Connection reset by peer (error 104)
2026-06-20T23:01:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:08 dist-test-slave-rdl9 krb5kdc[23133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:08 dist-test-slave-rdl9 krb5kdc[23133](info): set up 2 sockets
Jun 20 23:01:08 dist-test-slave-rdl9 krb5kdc[23133](info): commencing operation
krb5kdc: starting...
W20260620 23:01:09.578225 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:09 dist-test-slave-rdl9 krb5kdc[23133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:01:09.713275 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:09.807435 23146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:09.807812 23146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:09.807842 23146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:09.809280 23146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:09.809356 23146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:09.809370 23146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:09.809382 23146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:09.809394 23146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:09.811316 23146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:09.811638 23146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:09.811875 23146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:09.817543 23154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:09.817596 23151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:09.817559 23152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:09.817880 23146 server_base.cc:1034] running on GCE node
I20260620 23:01:09.818197 23146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:09.818537 23146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:09.819742 23146 hybrid_clock.cc:648] HybridClock initialized: now 1781996469819680 us; error 80 us; skew 500 ppm
Jun 20 23:01:09 dist-test-slave-rdl9 krb5kdc[23133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:09.822693 23146 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:01:09.823232 23146 webserver.cc:460] Webserver started at http://127.0.0.1:33271/ using document root <none> and password file <none>
I20260620 23:01:09.823491 23146 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:09.823549 23146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:09.823694 23146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:09.824862 23146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "093e7b8a49b54480a3fe61af4278d50b"
format_stamp: "Formatted at 2026-06-20 23:01:09 on dist-test-slave-rdl9"
server_key: "4f0e509cb868e9b5cdb5955d629eca79"
server_key_iv: "236c1089332903ece2fefffbeebd2878"
server_key_version: "encryptionkey@0"
I20260620 23:01:09.825234 23146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "093e7b8a49b54480a3fe61af4278d50b"
format_stamp: "Formatted at 2026-06-20 23:01:09 on dist-test-slave-rdl9"
server_key: "4f0e509cb868e9b5cdb5955d629eca79"
server_key_iv: "236c1089332903ece2fefffbeebd2878"
server_key_version: "encryptionkey@0"
I20260620 23:01:09.826946 23146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:09.827795 23161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:09.828034 23146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:09.828136 23146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "093e7b8a49b54480a3fe61af4278d50b"
format_stamp: "Formatted at 2026-06-20 23:01:09 on dist-test-slave-rdl9"
server_key: "4f0e509cb868e9b5cdb5955d629eca79"
server_key_iv: "236c1089332903ece2fefffbeebd2878"
server_key_version: "encryptionkey@0"
I20260620 23:01:09.828209 23146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:09.832058 23146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:09.834743 23146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:09.834916 23146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:09.838972 23146 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43999
I20260620 23:01:09.839231 23213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43999 every 8 connection(s)
I20260620 23:01:09.839383 23146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:09.840500 23214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:09.840544 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23146
I20260620 23:01:09.840652 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:09.840868 1688 external_mini_cluster.cc:1428] Setting key 65247ab69242c39fe79fbf7748b4e053
I20260620 23:01:09.843523 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:09.841446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:45077 (local address 127.0.0.1:43999)
0620 23:01:09.841653 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:09.841662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:09.841690 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:01:09.842164 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:09.842171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:09.842198 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:09.842328 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:09.843106 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:09.843110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:09.843110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:09.843114 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:09.843155 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:09.843166 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:09.843223 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:09.843295 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 23:01:09.843351 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260620 23:01:09.843947 23214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b: Bootstrap starting.
I20260620 23:01:09.844807 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:09.845098 23214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:09.845513 23214 log.cc:824] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:09.846529 23214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b: No bootstrap required, opened a new log
I20260620 23:01:09.847822 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:09.846694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.35:35787 (local address 127.0.0.1:43999)
0620 23:01:09.846742 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:09.846744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:09.846875 (+ 131us) server_negotiation.cc:407] Connection header received
0620 23:01:09.846935 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:09.846936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:09.846940 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:09.846985 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:09.847496 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:09.847497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:09.847498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:09.847501 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:09.847514 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:09.847518 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:09.847557 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:09.847760 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 23:01:09.847772 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:01:09.849529 23214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093e7b8a49b54480a3fe61af4278d50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } }
I20260620 23:01:09.849699 23214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:09.849731 23214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093e7b8a49b54480a3fe61af4278d50b, State: Initialized, Role: FOLLOWER
I20260620 23:01:09.849854 23214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093e7b8a49b54480a3fe61af4278d50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } }
I20260620 23:01:09.849941 23214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:09.849974 23214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:09.850025 23214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:09.850751 23214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093e7b8a49b54480a3fe61af4278d50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } }
I20260620 23:01:09.850862 23214 leader_election.cc:302] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 093e7b8a49b54480a3fe61af4278d50b; no voters:
I20260620 23:01:09.851047 23214 leader_election.cc:288] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:09.851162 23224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:09.851323 23214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:09.851395 23224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [term 1 LEADER]: Becoming Leader. State: Replica: 093e7b8a49b54480a3fe61af4278d50b, State: Running, Role: LEADER
I20260620 23:01:09.851552 23224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093e7b8a49b54480a3fe61af4278d50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } }
I20260620 23:01:09.852169 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 093e7b8a49b54480a3fe61af4278d50b. Latest consensus state: current_term: 1 leader_uuid: "093e7b8a49b54480a3fe61af4278d50b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093e7b8a49b54480a3fe61af4278d50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } } }
I20260620 23:01:09.852372 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:09.852397 23225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "093e7b8a49b54480a3fe61af4278d50b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "093e7b8a49b54480a3fe61af4278d50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } } }
I20260620 23:01:09.852478 23225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:09.853084 23234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:09.853688 23234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:09.855825 23234 catalog_manager.cc:1347] Generated new cluster ID: f1ea187e0c4b4d48aeb9344e6ff3deb7
I20260620 23:01:09.855919 23234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:09.897601 23234 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:09.897814 23234 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:09.907737 23234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 093e7b8a49b54480a3fe61af4278d50b: Generated new TSK 0
I20260620 23:01:09.907948 23234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:09.918992 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23146
2026-06-20T23:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:09 dist-test-slave-rdl9 krb5kdc[23245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:09 dist-test-slave-rdl9 krb5kdc[23245](info): set up 2 sockets
Jun 20 23:01:09 dist-test-slave-rdl9 krb5kdc[23245](info): commencing operation
krb5kdc: starting...
W20260620 23:01:10.882810 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:10 dist-test-slave-rdl9 krb5kdc[23245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:01:11.009861 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45291
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:41383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:45291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:11.102664 23258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:11.102908 23258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:11.102932 23258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:11.104175 23258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:11.104233 23258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:11.104246 23258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:11.104257 23258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:11.104269 23258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:11.105863 23258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:41383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:45291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:45291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:11.106127 23258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:11.106376 23258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:11.112169 23266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:11.112212 23264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:11.112219 23263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:11.112776 23258 server_base.cc:1034] running on GCE node
I20260620 23:01:11.113080 23258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:11.113689 23258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:11.114877 23258 hybrid_clock.cc:648] HybridClock initialized: now 1781996471114851 us; error 41 us; skew 500 ppm
Jun 20 23:01:11 dist-test-slave-rdl9 krb5kdc[23245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996471, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:11.121249 23258 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:01:11.121898 23258 webserver.cc:460] Webserver started at http://127.1.166.62:36073/ using document root <none> and password file <none>
I20260620 23:01:11.122165 23258 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:11.122220 23258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:11.122380 23258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:11.123499 23258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "8d60695032b0486b856bf5630bed610a"
format_stamp: "Formatted at 2026-06-20 23:01:11 on dist-test-slave-rdl9"
server_key: "9576d03d88d059c575f3e0adbe798517"
server_key_iv: "1266e3bbb39e3ce53a3f0b688a36adc4"
server_key_version: "encryptionkey@0"
I20260620 23:01:11.123842 23258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "8d60695032b0486b856bf5630bed610a"
format_stamp: "Formatted at 2026-06-20 23:01:11 on dist-test-slave-rdl9"
server_key: "9576d03d88d059c575f3e0adbe798517"
server_key_iv: "1266e3bbb39e3ce53a3f0b688a36adc4"
server_key_version: "encryptionkey@0"
I20260620 23:01:11.125478 23258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:11.126362 23273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:11.126571 23258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:11.126663 23258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "8d60695032b0486b856bf5630bed610a"
format_stamp: "Formatted at 2026-06-20 23:01:11 on dist-test-slave-rdl9"
server_key: "9576d03d88d059c575f3e0adbe798517"
server_key_iv: "1266e3bbb39e3ce53a3f0b688a36adc4"
server_key_version: "encryptionkey@0"
I20260620 23:01:11.126749 23258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:11.148065 23258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:11.150444 23258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:11.150574 23258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:11.154793 23258 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:45291
I20260620 23:01:11.154865 23325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:45291 every 8 connection(s)
I20260620 23:01:11.155212 23258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:11.155718 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23258
I20260620 23:01:11.155828 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:11.156003 1688 external_mini_cluster.cc:1428] Setting key bf5cfa17a2fa73ef5fd9ca879453af3d
I20260620 23:01:11.156314 23326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:11.159693 23326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a: Bootstrap starting.
I20260620 23:01:11.160451 23326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:11.160717 23326 log.cc:824] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:11.161437 23326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a: No bootstrap required, opened a new log
I20260620 23:01:11.163884 23326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d60695032b0486b856bf5630bed610a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45291 } }
Jun 20 23:01:11 dist-test-slave-rdl9 krb5kdc[23245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:11.164091 23326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:11.164125 23326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d60695032b0486b856bf5630bed610a, State: Initialized, Role: FOLLOWER
I20260620 23:01:11.164266 23326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d60695032b0486b856bf5630bed610a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45291 } }
I20260620 23:01:11.164356 23326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:11.164391 23326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:11.164435 23326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:11.165133 23326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d60695032b0486b856bf5630bed610a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45291 } }
I20260620 23:01:11.165238 23326 leader_election.cc:302] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d60695032b0486b856bf5630bed610a; no voters:
I20260620 23:01:11.165436 23326 leader_election.cc:288] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:11.165552 23331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:11.165709 23331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [term 1 LEADER]: Becoming Leader. State: Replica: 8d60695032b0486b856bf5630bed610a, State: Running, Role: LEADER
I20260620 23:01:11.165795 23326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:11.165899 23331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d60695032b0486b856bf5630bed610a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45291 } }
I20260620 23:01:11.166265 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d60695032b0486b856bf5630bed610a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d60695032b0486b856bf5630bed610a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45291 } } }
I20260620 23:01:11.166366 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:11.166599 23333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d60695032b0486b856bf5630bed610a. Latest consensus state: current_term: 1 leader_uuid: "8d60695032b0486b856bf5630bed610a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d60695032b0486b856bf5630bed610a" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 45291 } } }
I20260620 23:01:11.166726 23333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:11.166927 23338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:11.167474 23338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:11.169847 23338 catalog_manager.cc:1347] Generated new cluster ID: a41c26e2f6a74231ac48eeb40cb02434
I20260620 23:01:11.169885 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:11.156748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54779 (local address 127.1.166.62:45291)
0620 23:01:11.157044 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:11.157053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:11.157276 (+ 223us) server_negotiation.cc:407] Connection header received
0620 23:01:11.157803 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:11.157813 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:11.157893 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:11.158054 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:11.158773 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:11.160112 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:11.161304 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:11.161916 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:11.162038 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:11.164829 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:11.164837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:11.164837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:11.164859 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:11.167179 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:11.167838 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:11.167844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:11.167845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:11.168015 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:11.168349 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:11.168352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:11.168352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:11.168708 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:11.169261 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:11.169358 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 23:01:11.169608 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
I20260620 23:01:11.169917 23338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:11.171411 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:11.181982 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:11.173228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34765 (local address 127.1.166.62:45291)
0620 23:01:11.173289 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:11.173290 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:11.173355 (+ 65us) server_negotiation.cc:407] Connection header received
0620 23:01:11.173437 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:11.173438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:11.173449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:11.173493 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:11.174731 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:11.175475 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:11.176764 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:11.177123 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:11.177191 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:11.179066 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:11.179067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:11.179068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:11.179076 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:11.180679 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:11.181171 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:11.181172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:11.181173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:11.181229 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:11.181449 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:11.181449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:11.181450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:11.181576 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:11.181667 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:11.181817 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 23:01:11.181879 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:01:11.193393 23338 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:11.194665 23338 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:11.206934 23338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d60695032b0486b856bf5630bed610a: Generated new TSK 0
I20260620 23:01:11.207181 23338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:11.214214 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23258
W20260620 23:01:11.218473 23243 connection.cc:535] client connection to 127.1.166.62:45291 recv error: Network error: failed to read from TLS socket (remote: 127.1.166.62:45291): Connection reset by peer (error 104)
2026-06-20T23:01:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:11 dist-test-slave-rdl9 krb5kdc[23357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:11 dist-test-slave-rdl9 krb5kdc[23357](info): set up 2 sockets
Jun 20 23:01:11 dist-test-slave-rdl9 krb5kdc[23357](info): commencing operation
krb5kdc: starting...
W20260620 23:01:12.194514 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:12 dist-test-slave-rdl9 krb5kdc[23357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:01:12.325591 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:36805
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:38037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:36805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:12.413679 23370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:12.413944 23370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:12.413971 23370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:12.415417 23370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:12.415480 23370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:12.415494 23370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:12.415508 23370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:12.415518 23370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:12.417230 23370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:38037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:36805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:12.417521 23370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:12.417768 23370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:12.423147 23375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:12.423168 23378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:12.423183 23376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:12.423302 23370 server_base.cc:1034] running on GCE node
I20260620 23:01:12.423635 23370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:12.423950 23370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:12.425148 23370 hybrid_clock.cc:648] HybridClock initialized: now 1781996472425104 us; error 79 us; skew 500 ppm
Jun 20 23:01:12 dist-test-slave-rdl9 krb5kdc[23357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996472, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:12.427906 23370 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:01:12.428408 23370 webserver.cc:460] Webserver started at http://127.1.166.62:37173/ using document root <none> and password file <none>
I20260620 23:01:12.428633 23370 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:12.428683 23370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:12.428813 23370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:12.429888 23370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "512b93ce0a3545999124c4f7fa40f0a4"
format_stamp: "Formatted at 2026-06-20 23:01:12 on dist-test-slave-rdl9"
server_key: "50e798a3b3fecc612f7517c706937471"
server_key_iv: "37c25dddc869f8eb28d2fd891e01fb9b"
server_key_version: "encryptionkey@0"
I20260620 23:01:12.430552 23370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "512b93ce0a3545999124c4f7fa40f0a4"
format_stamp: "Formatted at 2026-06-20 23:01:12 on dist-test-slave-rdl9"
server_key: "50e798a3b3fecc612f7517c706937471"
server_key_iv: "37c25dddc869f8eb28d2fd891e01fb9b"
server_key_version: "encryptionkey@0"
I20260620 23:01:12.432361 23370 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:12.433434 23385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:12.433693 23370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:12.433807 23370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "512b93ce0a3545999124c4f7fa40f0a4"
format_stamp: "Formatted at 2026-06-20 23:01:12 on dist-test-slave-rdl9"
server_key: "50e798a3b3fecc612f7517c706937471"
server_key_iv: "37c25dddc869f8eb28d2fd891e01fb9b"
server_key_version: "encryptionkey@0"
I20260620 23:01:12.433902 23370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:12.450160 23370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:12.453006 23370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:12.453171 23370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:12.457886 23370 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:36805
I20260620 23:01:12.457935 23437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:36805 every 8 connection(s)
I20260620 23:01:12.458293 23370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:12.459280 23438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:12.461289 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23370
I20260620 23:01:12.461440 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:12.461642 1688 external_mini_cluster.cc:1428] Setting key 7acdb28999d4e64b055f3ded2cb95e5b
I20260620 23:01:12.463217 23438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4: Bootstrap starting.
I20260620 23:01:12.464085 23438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:12.464442 23438 log.cc:824] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:12.465590 23438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4: No bootstrap required, opened a new log
I20260620 23:01:12.468171 23438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512b93ce0a3545999124c4f7fa40f0a4" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36805 } }
I20260620 23:01:12.468365 23438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:12.468405 23438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512b93ce0a3545999124c4f7fa40f0a4, State: Initialized, Role: FOLLOWER
I20260620 23:01:12.468547 23438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512b93ce0a3545999124c4f7fa40f0a4" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36805 } }
I20260620 23:01:12.468626 23438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:12.468648 23438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:12.468683 23438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:12.468847 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:12.462627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51607 (local address 127.1.166.62:36805)
0620 23:01:12.462856 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:12.462866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:12.462904 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:01:12.463571 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:12.463580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:12.463643 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:12.463822 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:12.464124 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:12.465332 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:12.466583 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:12.467071 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:12.467179 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:12.468002 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:12.468006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:12.468006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:12.468193 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:12.468228 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:12.468240 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:12.468314 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:12.468447 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 23:01:12.468598 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260620 23:01:12.469689 23438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512b93ce0a3545999124c4f7fa40f0a4" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36805 } }
I20260620 23:01:12.469897 23438 leader_election.cc:302] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 512b93ce0a3545999124c4f7fa40f0a4; no voters:
I20260620 23:01:12.470193 23438 leader_election.cc:288] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:12.470269 23443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:12.470482 23443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [term 1 LEADER]: Becoming Leader. State: Replica: 512b93ce0a3545999124c4f7fa40f0a4, State: Running, Role: LEADER
I20260620 23:01:12.470480 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:12.470562 23438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:12.470593 23443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512b93ce0a3545999124c4f7fa40f0a4" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36805 } }
I20260620 23:01:12.470993 23444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "512b93ce0a3545999124c4f7fa40f0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512b93ce0a3545999124c4f7fa40f0a4" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36805 } } }
I20260620 23:01:12.470983 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 512b93ce0a3545999124c4f7fa40f0a4. Latest consensus state: current_term: 1 leader_uuid: "512b93ce0a3545999124c4f7fa40f0a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512b93ce0a3545999124c4f7fa40f0a4" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36805 } } }
I20260620 23:01:12.471064 23444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:12.471086 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:12.471303 23449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:12.471936 23449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:12.474224 23449 catalog_manager.cc:1347] Generated new cluster ID: bc86d9e6a49748b7846e2fdafcca11c6
I20260620 23:01:12.474294 23449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:12.477241 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:12.472868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58437 (local address 127.1.166.62:36805)
0620 23:01:12.472919 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:12.472922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:12.473240 (+ 318us) server_negotiation.cc:407] Connection header received
0620 23:01:12.473310 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:12.473312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:12.473327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:12.473434 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:12.473786 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:12.474590 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:12.475698 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:12.476040 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:12.476079 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:12.476969 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:12.476971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:12.476972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:12.476979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:12.476996 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:12.477001 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:12.477029 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:12.477135 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:01:12.477148 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:01:12.489795 23449 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:12.490379 23449 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:12.497030 23449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 512b93ce0a3545999124c4f7fa40f0a4: Generated new TSK 0
I20260620 23:01:12.497268 23449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:12.506211 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23370
W20260620 23:01:12.510934 23355 connection.cc:535] client connection to 127.1.166.62:36805 recv error: Network error: failed to read from TLS socket (remote: 127.1.166.62:36805): Connection reset by peer (error 104)
2026-06-20T23:01:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:12 dist-test-slave-rdl9 krb5kdc[23469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:12 dist-test-slave-rdl9 krb5kdc[23469](info): set up 2 sockets
Jun 20 23:01:12 dist-test-slave-rdl9 krb5kdc[23469](info): commencing operation
krb5kdc: starting...
W20260620 23:01:13.490621 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:13 dist-test-slave-rdl9 krb5kdc[23469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:01:13.613778 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:44169
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:44169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:13.702652 23482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:13.702893 23482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:13.702916 23482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:13.704373 23482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:13.704487 23482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:13.704504 23482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:13.704516 23482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:13.704530 23482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:13.707082 23482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:44169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:44169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:13.707358 23482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:13.707549 23482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:13.713263 23490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:13.713384 23488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:13.713323 23487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:13.713760 23482 server_base.cc:1034] running on GCE node
I20260620 23:01:13.713994 23482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:13.714285 23482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:13.715471 23482 hybrid_clock.cc:648] HybridClock initialized: now 1781996473715445 us; error 56 us; skew 500 ppm
Jun 20 23:01:13 dist-test-slave-rdl9 krb5kdc[23469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996473, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:13.718240 23482 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:01:13.718796 23482 webserver.cc:460] Webserver started at http://127.1.166.62:39569/ using document root <none> and password file <none>
I20260620 23:01:13.719022 23482 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:13.719069 23482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:13.719200 23482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:13.720429 23482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "dd5de7cbb532453a94e25c9afaebf724"
format_stamp: "Formatted at 2026-06-20 23:01:13 on dist-test-slave-rdl9"
server_key: "ff1b2d1e56b838ae97f141cebfba4692"
server_key_iv: "dee030b9c1466adfd6c7782e5024faf1"
server_key_version: "encryptionkey@0"
I20260620 23:01:13.720801 23482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "dd5de7cbb532453a94e25c9afaebf724"
format_stamp: "Formatted at 2026-06-20 23:01:13 on dist-test-slave-rdl9"
server_key: "ff1b2d1e56b838ae97f141cebfba4692"
server_key_iv: "dee030b9c1466adfd6c7782e5024faf1"
server_key_version: "encryptionkey@0"
I20260620 23:01:13.722414 23482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:13.723665 23497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:13.723927 23482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:13.724025 23482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "dd5de7cbb532453a94e25c9afaebf724"
format_stamp: "Formatted at 2026-06-20 23:01:13 on dist-test-slave-rdl9"
server_key: "ff1b2d1e56b838ae97f141cebfba4692"
server_key_iv: "dee030b9c1466adfd6c7782e5024faf1"
server_key_version: "encryptionkey@0"
I20260620 23:01:13.724109 23482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:13.727825 23482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:13.730814 23482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:13.730962 23482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:13.735678 23482 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:44169
I20260620 23:01:13.735734 23549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:44169 every 8 connection(s)
I20260620 23:01:13.736086 23482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:13.737164 23550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:13.739705 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23482
I20260620 23:01:13.739842 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:13.739825 23550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724: Bootstrap starting.
I20260620 23:01:13.740008 1688 external_mini_cluster.cc:1428] Setting key d53107347c921284bddb6be495906cb8
I20260620 23:01:13.740576 23550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:13.740931 23550 log.cc:824] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:13.741775 23550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724: No bootstrap required, opened a new log
I20260620 23:01:13.743463 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:13.740635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53741 (local address 127.1.166.62:44169)
0620 23:01:13.740868 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:13.740879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:13.740920 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:01:13.741555 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:13.741566 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:13.741608 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:13.741824 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:13.742864 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:13.742868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:13.742868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:13.742874 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:13.742928 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:13.742945 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:13.743059 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:13.743177 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 23:01:13.743227 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
I20260620 23:01:13.744606 23550 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5de7cbb532453a94e25c9afaebf724" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44169 } }
I20260620 23:01:13.744789 23550 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:13.744813 23550 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5de7cbb532453a94e25c9afaebf724, State: Initialized, Role: FOLLOWER
I20260620 23:01:13.744951 23550 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5de7cbb532453a94e25c9afaebf724" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44169 } }
I20260620 23:01:13.745059 23550 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:13.745061 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:13.745097 23550 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:13.745155 23550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:13.746021 23550 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5de7cbb532453a94e25c9afaebf724" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44169 } }
I20260620 23:01:13.746181 23550 leader_election.cc:302] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd5de7cbb532453a94e25c9afaebf724; no voters:
I20260620 23:01:13.746501 23550 leader_election.cc:288] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:13.746526 23559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:13.746848 23550 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:13.747148 23559 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [term 1 LEADER]: Becoming Leader. State: Replica: dd5de7cbb532453a94e25c9afaebf724, State: Running, Role: LEADER
I20260620 23:01:13.747293 23559 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5de7cbb532453a94e25c9afaebf724" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44169 } }
I20260620 23:01:13.747823 23562 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd5de7cbb532453a94e25c9afaebf724. Latest consensus state: current_term: 1 leader_uuid: "dd5de7cbb532453a94e25c9afaebf724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5de7cbb532453a94e25c9afaebf724" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44169 } } }
I20260620 23:01:13.748071 23560 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd5de7cbb532453a94e25c9afaebf724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd5de7cbb532453a94e25c9afaebf724" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 44169 } } }
I20260620 23:01:13.748121 23562 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:13.748199 23560 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:13.748476 23571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:13.748588 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:13.746912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51793 (local address 127.1.166.62:44169)
0620 23:01:13.746999 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:13.747002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:13.747109 (+ 107us) server_negotiation.cc:407] Connection header received
0620 23:01:13.747234 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:13.747236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:13.747243 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:13.747278 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:13.748060 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:13.748061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:13.748061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:13.748066 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:13.748083 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:13.748088 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:13.748141 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:13.748509 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:01:13.748525 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:01:13.749257 23571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:13.751379 23571 catalog_manager.cc:1347] Generated new cluster ID: 817ad7e9eee847f0b00c88b3f6e18f42
I20260620 23:01:13.751454 23571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:13.772593 23571 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:13.772786 23571 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:13.776280 23571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd5de7cbb532453a94e25c9afaebf724: Generated new TSK 0
I20260620 23:01:13.776477 23571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:13.786129 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23482
2026-06-20T23:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:13 dist-test-slave-rdl9 krb5kdc[23581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:13 dist-test-slave-rdl9 krb5kdc[23581](info): set up 2 sockets
Jun 20 23:01:13 dist-test-slave-rdl9 krb5kdc[23581](info): commencing operation
krb5kdc: starting...
W20260620 23:01:14.766270 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:14 dist-test-slave-rdl9 krb5kdc[23581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:01:14.889567 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:36823
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:46219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:36823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:14.975791 23594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:14.976035 23594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:14.976060 23594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:14.977383 23594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:14.977444 23594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:14.977458 23594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:14.977469 23594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:14.977479 23594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:14.979176 23594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:46219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:36823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:36823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:14.979442 23594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:14.979663 23594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:01:14.984704 23594 server_base.cc:1034] running on GCE node
W20260620 23:01:14.984681 23600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:14.984992 23602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:14.984790 23599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:14.985450 23594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:14.985831 23594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:14.987017 23594 hybrid_clock.cc:648] HybridClock initialized: now 1781996474986966 us; error 70 us; skew 500 ppm
Jun 20 23:01:14 dist-test-slave-rdl9 krb5kdc[23581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996474, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:14.989779 23594 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:01:14.990283 23594 webserver.cc:460] Webserver started at http://127.1.166.62:43079/ using document root <none> and password file <none>
I20260620 23:01:14.990526 23594 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:14.990741 23594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:14.990895 23594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:14.992090 23594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "852038cdefa9429b9a6e3504061188f8"
format_stamp: "Formatted at 2026-06-20 23:01:14 on dist-test-slave-rdl9"
server_key: "248a7d8463cbc1e41beba2a256c3cac0"
server_key_iv: "235f95164e7a7d1a7dd29d8b83a9e32a"
server_key_version: "encryptionkey@0"
I20260620 23:01:14.992465 23594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "852038cdefa9429b9a6e3504061188f8"
format_stamp: "Formatted at 2026-06-20 23:01:14 on dist-test-slave-rdl9"
server_key: "248a7d8463cbc1e41beba2a256c3cac0"
server_key_iv: "235f95164e7a7d1a7dd29d8b83a9e32a"
server_key_version: "encryptionkey@0"
I20260620 23:01:14.993880 23594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:14.994784 23609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:14.994984 23594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:14.995081 23594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "852038cdefa9429b9a6e3504061188f8"
format_stamp: "Formatted at 2026-06-20 23:01:14 on dist-test-slave-rdl9"
server_key: "248a7d8463cbc1e41beba2a256c3cac0"
server_key_iv: "235f95164e7a7d1a7dd29d8b83a9e32a"
server_key_version: "encryptionkey@0"
I20260620 23:01:14.995172 23594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:15.037214 23594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:15.039916 23594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:15.040064 23594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:15.044004 23594 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:36823
I20260620 23:01:15.044132 23661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:36823 every 8 connection(s)
I20260620 23:01:15.044412 23594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:15.045347 23662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:15.045821 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23594
I20260620 23:01:15.045956 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:15.046249 1688 external_mini_cluster.cc:1428] Setting key 0ea057ae49e1ebce31c188887ce9e0ea
I20260620 23:01:15.048614 23662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8: Bootstrap starting.
I20260620 23:01:15.049386 23662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:15.049722 23662 log.cc:824] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:15.050763 23662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8: No bootstrap required, opened a new log
I20260620 23:01:15.052938 23662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852038cdefa9429b9a6e3504061188f8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36823 } }
I20260620 23:01:15.053112 23662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:15.053133 23662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852038cdefa9429b9a6e3504061188f8, State: Initialized, Role: FOLLOWER
I20260620 23:01:15.053247 23662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852038cdefa9429b9a6e3504061188f8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36823 } }
I20260620 23:01:15.053324 23662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:15.053360 23662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:15.053411 23662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:15.054256 23662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852038cdefa9429b9a6e3504061188f8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36823 } }
Jun 20 23:01:15 dist-test-slave-rdl9 krb5kdc[23581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:15.054430 23662 leader_election.cc:302] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 852038cdefa9429b9a6e3504061188f8; no voters:
I20260620 23:01:15.054651 23662 leader_election.cc:288] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:15.054718 23667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:15.054939 23662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:15.054967 23667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [term 1 LEADER]: Becoming Leader. State: Replica: 852038cdefa9429b9a6e3504061188f8, State: Running, Role: LEADER
I20260620 23:01:15.055137 23667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852038cdefa9429b9a6e3504061188f8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36823 } }
I20260620 23:01:15.055567 23668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "852038cdefa9429b9a6e3504061188f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852038cdefa9429b9a6e3504061188f8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36823 } } }
I20260620 23:01:15.055651 23668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:15.055869 23669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 852038cdefa9429b9a6e3504061188f8. Latest consensus state: current_term: 1 leader_uuid: "852038cdefa9429b9a6e3504061188f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852038cdefa9429b9a6e3504061188f8" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 36823 } } }
I20260620 23:01:15.055985 23669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:15.056151 23676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:15.056967 23676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:15.059264 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:15.047000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47051 (local address 127.1.166.62:36823)
0620 23:01:15.047208 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:15.047219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:15.047259 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:01:15.047797 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:15.047806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:15.047885 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:15.048040 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:15.048649 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:15.050172 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:15.051208 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:15.051766 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:15.051879 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:15.055099 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:15.055105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:15.055105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:15.055120 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:15.057206 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:15.057710 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:15.057715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:15.057716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:15.057892 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:15.058123 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:15.058124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:15.058124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:15.058530 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:15.058721 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:15.058829 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:01:15.059003 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260620 23:01:15.059628 23676 catalog_manager.cc:1347] Generated new cluster ID: 1a8d319348cb4634baa376dd2959f5df
I20260620 23:01:15.059688 23676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:15.060421 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:15.075676 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:15.062321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53157 (local address 127.1.166.62:36823)
0620 23:01:15.062398 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:15.062400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:15.066708 (+ 4308us) server_negotiation.cc:407] Connection header received
0620 23:01:15.066797 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:15.066798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:15.066813 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:15.066918 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:15.067930 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:15.068721 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:15.069874 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:15.070237 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:15.070324 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:15.071919 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:15.071921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:15.071921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:15.071929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:15.073621 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:15.074078 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:15.074079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:15.074079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:15.074143 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:15.074333 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:15.074334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:15.074334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:15.075010 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:15.075201 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:15.075408 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 23:01:15.075551 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:01:15.081636 23676 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:15.082506 23676 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:15.089303 23676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 852038cdefa9429b9a6e3504061188f8: Generated new TSK 0
I20260620 23:01:15.089550 23676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:15.100708 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23594
2026-06-20T23:01:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:15 dist-test-slave-rdl9 krb5kdc[23693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:15 dist-test-slave-rdl9 krb5kdc[23693](info): set up 2 sockets
Jun 20 23:01:15 dist-test-slave-rdl9 krb5kdc[23693](info): commencing operation
krb5kdc: starting...
W20260620 23:01:16.074735 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:16 dist-test-slave-rdl9 krb5kdc[23693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:01:16.201615 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:37003
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:37003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:16.291360 23706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:16.291800 23706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:16.291929 23706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:16.295938 23706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:16.296041 23706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:16.296062 23706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:16.296083 23706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:16.296103 23706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:16.300503 23706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:37003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:16.300911 23706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:16.301234 23706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:16.306731 23711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:16.306733 23712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:16.307013 23714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:16.307977 23706 server_base.cc:1034] running on GCE node
I20260620 23:01:16.308334 23706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:16.308697 23706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:16.309849 23706 hybrid_clock.cc:648] HybridClock initialized: now 1781996476309822 us; error 42 us; skew 500 ppm
Jun 20 23:01:16 dist-test-slave-rdl9 krb5kdc[23693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996476, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:16.313918 23706 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:01:16.314690 23706 webserver.cc:460] Webserver started at http://127.1.166.62:42529/ using document root <none> and password file <none>
I20260620 23:01:16.315026 23706 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:16.315101 23706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:16.315260 23706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:16.317028 23706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "318e06ca16984a2896188eb7ef6e656c"
format_stamp: "Formatted at 2026-06-20 23:01:16 on dist-test-slave-rdl9"
server_key: "73d5592648d732c1141fe38473cbc689"
server_key_iv: "543baa1cb765eb46bb0b6f42c58c5b1d"
server_key_version: "encryptionkey@0"
I20260620 23:01:16.317526 23706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "318e06ca16984a2896188eb7ef6e656c"
format_stamp: "Formatted at 2026-06-20 23:01:16 on dist-test-slave-rdl9"
server_key: "73d5592648d732c1141fe38473cbc689"
server_key_iv: "543baa1cb765eb46bb0b6f42c58c5b1d"
server_key_version: "encryptionkey@0"
I20260620 23:01:16.319576 23706 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 23:01:16.320720 23721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:16.320950 23706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:16.321061 23706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "318e06ca16984a2896188eb7ef6e656c"
format_stamp: "Formatted at 2026-06-20 23:01:16 on dist-test-slave-rdl9"
server_key: "73d5592648d732c1141fe38473cbc689"
server_key_iv: "543baa1cb765eb46bb0b6f42c58c5b1d"
server_key_version: "encryptionkey@0"
I20260620 23:01:16.321152 23706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:16.335110 23706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:16.338228 23706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:16.338477 23706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:16.344488 23706 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:37003
I20260620 23:01:16.344592 23773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:37003 every 8 connection(s)
I20260620 23:01:16.344916 23706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:16.346025 23774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:16.348176 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23706
I20260620 23:01:16.348309 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:16.348501 1688 external_mini_cluster.cc:1428] Setting key 59ff730c62fd18eb3e35c9ae59e1eca3
I20260620 23:01:16.349427 23774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c: Bootstrap starting.
I20260620 23:01:16.350143 23774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:16.350478 23774 log.cc:824] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:16.351392 23774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c: No bootstrap required, opened a new log
I20260620 23:01:16.353744 23774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318e06ca16984a2896188eb7ef6e656c" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 37003 } }
I20260620 23:01:16.353924 23774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:16.353948 23774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318e06ca16984a2896188eb7ef6e656c, State: Initialized, Role: FOLLOWER
I20260620 23:01:16.354097 23774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318e06ca16984a2896188eb7ef6e656c" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 37003 } }
I20260620 23:01:16.354162 23774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:16.354187 23774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:16.354233 23774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:16.355079 23774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318e06ca16984a2896188eb7ef6e656c" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 37003 } }
I20260620 23:01:16.355201 23774 leader_election.cc:302] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 318e06ca16984a2896188eb7ef6e656c; no voters:
I20260620 23:01:16.355393 23774 leader_election.cc:288] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:16.355466 23779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:16.355705 23774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:16.355808 23779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [term 1 LEADER]: Becoming Leader. State: Replica: 318e06ca16984a2896188eb7ef6e656c, State: Running, Role: LEADER
I20260620 23:01:16.355940 23779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318e06ca16984a2896188eb7ef6e656c" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 37003 } }
I20260620 23:01:16.356455 23779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318e06ca16984a2896188eb7ef6e656c. Latest consensus state: current_term: 1 leader_uuid: "318e06ca16984a2896188eb7ef6e656c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318e06ca16984a2896188eb7ef6e656c" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 37003 } } }
I20260620 23:01:16.356564 23779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:16.356463 23780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "318e06ca16984a2896188eb7ef6e656c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "318e06ca16984a2896188eb7ef6e656c" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 37003 } } }
I20260620 23:01:16.356626 23780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:16.356812 23788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:16.357156 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:16.349058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49435 (local address 127.1.166.62:37003)
0620 23:01:16.349259 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:16.349267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:16.349293 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:01:16.350517 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:16.350531 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:16.350626 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:16.350803 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:16.351363 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:16.352568 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:16.353799 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:16.354435 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:16.354547 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:16.355312 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:16.355319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:16.355319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:16.355689 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:16.355761 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:16.355781 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:16.356233 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:16.356275 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 23:01:16.356370 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260620 23:01:16.357529 23788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:16.358489 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:16.359583 23788 catalog_manager.cc:1347] Generated new cluster ID: f6f4a9616edb48578621df6476f53674
I20260620 23:01:16.359704 23788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:16.364434 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:16.360185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42113 (local address 127.1.166.62:37003)
0620 23:01:16.360252 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:16.360255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:16.360369 (+ 114us) server_negotiation.cc:407] Connection header received
0620 23:01:16.360414 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:16.360416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:16.360431 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:16.360515 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:16.360690 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:16.361601 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:16.362991 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:16.363311 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:16.363398 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:16.364137 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:16.364139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:16.364139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:16.364147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:16.364166 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:16.364171 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:16.364233 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:16.364276 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 23:01:16.364311 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:01:16.371062 23788 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:16.371891 23788 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:16.374823 23788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 318e06ca16984a2896188eb7ef6e656c: Generated new TSK 0
I20260620 23:01:16.375036 23788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:16.377173 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23706
2026-06-20T23:01:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:16 dist-test-slave-rdl9 krb5kdc[23805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:16 dist-test-slave-rdl9 krb5kdc[23805](info): set up 2 sockets
Jun 20 23:01:16 dist-test-slave-rdl9 krb5kdc[23805](info): commencing operation
krb5kdc: starting...
W20260620 23:01:17.362843 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:17 dist-test-slave-rdl9 krb5kdc[23805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:01:17.485080 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:41795
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:17.579330 23818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:17.579739 23818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:17.579803 23818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:17.581789 23818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:17.581852 23818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:17.581866 23818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:17.581878 23818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:17.581890 23818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:17.583770 23818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:17.584044 23818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:17.584275 23818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:17.590306 23826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:17.590306 23823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:17.590642 23824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:17.591439 23818 server_base.cc:1034] running on GCE node
I20260620 23:01:17.591898 23818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:17.592358 23818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:17.593528 23818 hybrid_clock.cc:648] HybridClock initialized: now 1781996477593485 us; error 47 us; skew 500 ppm
Jun 20 23:01:17 dist-test-slave-rdl9 krb5kdc[23805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996477, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:17.596733 23818 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:01:17.597200 23818 webserver.cc:460] Webserver started at http://127.1.166.62:37101/ using document root <none> and password file <none>
I20260620 23:01:17.597411 23818 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:17.597460 23818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:17.597587 23818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:17.598845 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "b1c6f20905804342beb188aa1d020738"
format_stamp: "Formatted at 2026-06-20 23:01:17 on dist-test-slave-rdl9"
server_key: "9697f8ca6d495f70aa477a63d469c933"
server_key_iv: "d4c54263ca09c1dc2751e4943cab969d"
server_key_version: "encryptionkey@0"
I20260620 23:01:17.599460 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "b1c6f20905804342beb188aa1d020738"
format_stamp: "Formatted at 2026-06-20 23:01:17 on dist-test-slave-rdl9"
server_key: "9697f8ca6d495f70aa477a63d469c933"
server_key_iv: "d4c54263ca09c1dc2751e4943cab969d"
server_key_version: "encryptionkey@0"
I20260620 23:01:17.600996 23818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:17.601826 23833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:17.602039 23818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:17.602133 23818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "b1c6f20905804342beb188aa1d020738"
format_stamp: "Formatted at 2026-06-20 23:01:17 on dist-test-slave-rdl9"
server_key: "9697f8ca6d495f70aa477a63d469c933"
server_key_iv: "d4c54263ca09c1dc2751e4943cab969d"
server_key_version: "encryptionkey@0"
I20260620 23:01:17.602200 23818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:17.605907 23818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:17.608105 23818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:17.608269 23818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:17.614471 23885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:41795 every 8 connection(s)
I20260620 23:01:17.614480 23818 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:41795
I20260620 23:01:17.615394 23818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:17.617316 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:17.620438 23886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738: Bootstrap starting.
I20260620 23:01:17.620821 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23818
I20260620 23:01:17.620932 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:17.621119 1688 external_mini_cluster.cc:1428] Setting key bcbdd2e04763755a806d5049fe43e319
I20260620 23:01:17.621205 23886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:17.621496 23886 log.cc:824] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:17.622301 23886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738: No bootstrap required, opened a new log
I20260620 23:01:17.624133 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:17.621672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36275 (local address 127.1.166.62:41795)
0620 23:01:17.621896 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:17.621907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:17.621942 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:01:17.622516 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:17.622523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:17.622552 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:17.622689 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:17.623460 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:17.623464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:17.623465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:17.623469 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:17.623508 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:17.623519 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:17.623592 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:17.623647 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:01:17.623691 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
I20260620 23:01:17.624027 23886 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c6f20905804342beb188aa1d020738" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41795 } }
I20260620 23:01:17.624200 23886 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:17.624236 23886 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c6f20905804342beb188aa1d020738, State: Initialized, Role: FOLLOWER
I20260620 23:01:17.624379 23886 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c6f20905804342beb188aa1d020738" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41795 } }
I20260620 23:01:17.624495 23886 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:17.624536 23886 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:17.624583 23886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:17.625144 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:17.625381 23886 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c6f20905804342beb188aa1d020738" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41795 } }
I20260620 23:01:17.625489 23886 leader_election.cc:302] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1c6f20905804342beb188aa1d020738; no voters:
I20260620 23:01:17.625679 23886 leader_election.cc:288] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:17.625715 23891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:17.625833 23891 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [term 1 LEADER]: Becoming Leader. State: Replica: b1c6f20905804342beb188aa1d020738, State: Running, Role: LEADER
I20260620 23:01:17.625931 23891 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c6f20905804342beb188aa1d020738" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41795 } }
I20260620 23:01:17.625968 23886 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:17.626405 23893 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1c6f20905804342beb188aa1d020738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c6f20905804342beb188aa1d020738" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41795 } } }
I20260620 23:01:17.626484 23893 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:17.626279 23891 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1c6f20905804342beb188aa1d020738. Latest consensus state: current_term: 1 leader_uuid: "b1c6f20905804342beb188aa1d020738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c6f20905804342beb188aa1d020738" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41795 } } }
I20260620 23:01:17.626665 23891 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738 [sys.catalog]: This master's current role is: LEADER
W20260620 23:01:17.627895 23911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:17.627969 23911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:17.628026 23903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:17.628273 23903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:17.629535 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:17.628073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41481 (local address 127.1.166.62:41795)
0620 23:01:17.628132 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:17.628135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:17.628348 (+ 213us) server_negotiation.cc:407] Connection header received
0620 23:01:17.628423 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:17.628424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:17.628433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:17.628484 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:17.629243 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:17.629244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:17.629245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:17.629250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:17.629272 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:17.629277 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:17.629342 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:17.629429 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:01:17.629448 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:01:17.630476 23903 catalog_manager.cc:1347] Generated new cluster ID: 09666c1fbcbb473f97bc25208f547e5b
I20260620 23:01:17.630537 23903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:17.637982 23903 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:17.638227 23903 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:17.648695 23903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1c6f20905804342beb188aa1d020738: Generated new TSK 0
I20260620 23:01:17.648928 23903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:17.653644 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23818
2026-06-20T23:01:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1277 ms)
[----------] 18 tests from AuthTokenIssuingTest (23238 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:17 dist-test-slave-rdl9 krb5kdc[23917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:17 dist-test-slave-rdl9 krb5kdc[23917](info): set up 2 sockets
Jun 20 23:01:17 dist-test-slave-rdl9 krb5kdc[23917](info): commencing operation
krb5kdc: starting...
W20260620 23:01:18.646795 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:18 dist-test-slave-rdl9 krb5kdc[23917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:01:18.775161 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:41427
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:41427,127.1.166.61:36575,127.1.166.60:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:18.861469 23930 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:01:18.861709 23930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:18.861735 23930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:18.861757 23930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:18.862931 23930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:18.862972 23930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:18.862984 23930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:18.862995 23930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:18.863008 23930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:18.864503 23930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:41427,127.1.166.61:36575,127.1.166.60:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:41427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:18.864740 23930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:18.864931 23930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:18.870010 23935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:18.870000 23938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:18.870000 23936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:18.870577 23930 server_base.cc:1034] running on GCE node
I20260620 23:01:18.870947 23930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:18.871284 23930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:18.872459 23930 hybrid_clock.cc:648] HybridClock initialized: now 1781996478872417 us; error 65 us; skew 500 ppm
Jun 20 23:01:18 dist-test-slave-rdl9 krb5kdc[23917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996478, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:18.875101 23930 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:01:18.875547 23930 webserver.cc:460] Webserver started at http://127.1.166.62:41043/ using document root <none> and password file <none>
I20260620 23:01:18.875761 23930 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:18.875809 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:18.875942 23930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:18.877110 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "1888da909a6f4fb995bb31718c636f33"
format_stamp: "Formatted at 2026-06-20 23:01:18 on dist-test-slave-rdl9"
server_key: "2d39617ef6ba96df668a54ccae461efc"
server_key_iv: "04a70355ca3b403dc256bae300640039"
server_key_version: "encryptionkey@0"
I20260620 23:01:18.877475 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "1888da909a6f4fb995bb31718c636f33"
format_stamp: "Formatted at 2026-06-20 23:01:18 on dist-test-slave-rdl9"
server_key: "2d39617ef6ba96df668a54ccae461efc"
server_key_iv: "04a70355ca3b403dc256bae300640039"
server_key_version: "encryptionkey@0"
I20260620 23:01:18.879004 23930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:18.879853 23945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:18.880041 23930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:18.880132 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "1888da909a6f4fb995bb31718c636f33"
format_stamp: "Formatted at 2026-06-20 23:01:18 on dist-test-slave-rdl9"
server_key: "2d39617ef6ba96df668a54ccae461efc"
server_key_iv: "04a70355ca3b403dc256bae300640039"
server_key_version: "encryptionkey@0"
I20260620 23:01:18.880198 23930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:18.892946 23930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:18.895390 23930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:18.895540 23930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:18.899820 23930 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:41427
I20260620 23:01:18.899917 23997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:41427 every 8 connection(s)
I20260620 23:01:18.900267 23930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:18.900543 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 23930
I20260620 23:01:18.900651 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:18.900856 1688 external_mini_cluster.cc:1428] Setting key 07134b54dc90bcf54ca07ee6846c34d6
I20260620 23:01:18.901476 23998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:18.903386 23998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.166.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.61@KRBTEST.COM" created.
Jun 20 23:01:18 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996478, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:18.915424 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:18.904303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.1.166.62:41427)
0620 23:01:18.904457 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:18.904462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:18.904530 (+ 68us) server_negotiation.cc:407] Connection header received
0620 23:01:18.905033 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:18.905036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:18.905059 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:18.905177 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:18.905786 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:18.907244 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:18.908199 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:18.908704 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:18.908828 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:18.912227 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:18.912234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:18.912235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:18.912242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:18.913865 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:18.914242 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:18.914245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:18.914245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:18.914322 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:18.914546 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:18.914547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:18.914547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:18.914788 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:18.914864 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:18.915044 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:01:18.915162 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260620 23:01:18.915426 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:18.904088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41427 (local address 127.0.0.1:46802)
0620 23:01:18.904278 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:18.904322 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:18.904864 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:18.905234 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:18.905236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:18.905505 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:18.905674 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:18.905678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:18.907377 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:18.907378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:18.908106 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:18.908108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:18.908832 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:18.908833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:18.908928 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:18.909523 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:18.909532 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:18.912132 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:18.913968 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:18.913974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:18.913977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:18.914166 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:18.914394 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:18.914395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:18.914396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:18.914489 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:18.914852 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:18.914858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:18.914923 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 23:01:18.915006 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:01:18.915242 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":79,"thread_start_us":80,"threads_started":1}
I20260620 23:01:18.916540 23998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:18.916908 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:18.916791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58152)
0620 23:01:18.916836 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:18.916881 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 23:01:18.917043 23947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
I20260620 23:01:18.917436 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:18.917351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58156)
0620 23:01:18.917388 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:18.917412 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 23:01:18.917529 23998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:36575: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab.
Entry for principal kudu/127.1.166.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab.
I20260620 23:01:18.926959 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.61:36575
--webserver_interface=127.1.166.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab
--principal=kudu/127.1.166.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:41427,127.1.166.61:36575,127.1.166.60:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:01:18.966768 23998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } attempt: 1
I20260620 23:01:18.967439 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:18.967182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58160)
0620 23:01:18.967281 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:18.967376 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:01:18.967960 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:18.967853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58170)
0620 23:01:18.967902 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:18.967936 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 23:01:18.968127 23998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:36575: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
W20260620 23:01:19.016953 24005 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:01:19.017196 24005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:19.017215 24005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:19.017241 24005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:19.018508 24005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:19.018559 24005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:19.018574 24005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:19.018585 24005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:19.018597 24005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:19.020159 24005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:41427,127.1.166.61:36575,127.1.166.60:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab
--principal=kudu/127.1.166.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.61:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.166.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:19.020392 24005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:19.020601 24005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:19.025185 24013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:19.025349 24005 server_base.cc:1034] running on GCE node
W20260620 23:01:19.025178 24010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:19.025206 24011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:19.025664 24005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:19.025959 24005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:19.027140 24005 hybrid_clock.cc:648] HybridClock initialized: now 1781996479027084 us; error 69 us; skew 500 ppm
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:19.029742 24005 init.cc:377] Logged in from keytab as kudu/127.1.166.61@KRBTEST.COM (short username kudu)
I20260620 23:01:19.030189 24005 webserver.cc:460] Webserver started at http://127.1.166.61:36329/ using document root <none> and password file <none>
I20260620 23:01:19.030457 24005 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:19.030510 24005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:19.030632 24005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:19.031751 24005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/data/instance:
uuid: "ba18fdbe5b964e129736da755205d4b2"
format_stamp: "Formatted at 2026-06-20 23:01:19 on dist-test-slave-rdl9"
server_key: "04e43303d92c49c7afffc739921b9876"
server_key_iv: "5b6eb800d91f893309c3cf7a22a6e23d"
server_key_version: "encryptionkey@0"
I20260620 23:01:19.032121 24005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/wal/instance:
uuid: "ba18fdbe5b964e129736da755205d4b2"
format_stamp: "Formatted at 2026-06-20 23:01:19 on dist-test-slave-rdl9"
server_key: "04e43303d92c49c7afffc739921b9876"
server_key_iv: "5b6eb800d91f893309c3cf7a22a6e23d"
server_key_version: "encryptionkey@0"
I20260620 23:01:19.033737 24005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:19.034888 24020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:19.035230 24005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:19.035348 24005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/wal
uuid: "ba18fdbe5b964e129736da755205d4b2"
format_stamp: "Formatted at 2026-06-20 23:01:19 on dist-test-slave-rdl9"
server_key: "04e43303d92c49c7afffc739921b9876"
server_key_iv: "5b6eb800d91f893309c3cf7a22a6e23d"
server_key_version: "encryptionkey@0"
I20260620 23:01:19.035445 24005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:19.036312 23998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } attempt: 2
I20260620 23:01:19.036767 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.036643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58178)
0620 23:01:19.036699 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.036731 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 23:01:19.037313 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.037205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58188)
0620 23:01:19.037257 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.037289 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 23:01:19.037417 23998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:36575: Network error: Client connection negotiation failed: client connection to 127.1.166.61:36575: connect: Connection refused (error 111)
I20260620 23:01:19.046504 24005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:19.048693 24005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:19.048843 24005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:19.053681 24005 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.61:36575
I20260620 23:01:19.053841 24072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.61:36575 every 8 connection(s)
I20260620 23:01:19.054124 24005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
I20260620 23:01:19.055318 24073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:19.056977 24073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:19.062538 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 24005
I20260620 23:01:19.062638 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-1/wal/instance
I20260620 23:01:19.062848 1688 external_mini_cluster.cc:1428] Setting key 2ece1929f30663ed85d5ed13b831b25c
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:19.067708 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.057924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.1.166.62:41427)
0620 23:01:19.057975 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.057977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.058244 (+ 267us) server_negotiation.cc:407] Connection header received
0620 23:01:19.058952 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.058953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.058968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.059059 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.059899 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.060482 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.061670 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.061913 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.061993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.064655 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.064660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.064660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.064674 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.066249 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.066724 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.066726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.066726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.066781 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.067025 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.067025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.067026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.067157 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.067212 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.067509 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.067609 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:01:19.067884 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.057863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41427 (local address 127.0.0.1:46814)
0620 23:01:19.058089 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.058141 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:19.058822 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:19.059207 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:19.059212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:19.059520 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:19.059750 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.059755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.060627 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.060628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.061536 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.061539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.062035 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.062036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.062184 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.062806 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:19.062818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:19.064463 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:19.066394 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.066404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.066407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.066650 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.066837 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.066837 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.066838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.066955 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.067214 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:19.067220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:19.067375 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:01:19.067464 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:01:19.067649 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260620 23:01:19.068470 24073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.166.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.60@KRBTEST.COM" created.
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
I20260620 23:01:19.077021 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.068728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58198)
0620 23:01:19.068769 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.068780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:19.068821 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:19.069103 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:19.069104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:19.069260 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:19.069344 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.069346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.070456 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.070458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.071422 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.071426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.071996 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.071997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.072067 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.072651 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:19.072655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:19.074070 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:19.076053 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.076054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.076054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.076204 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.076389 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.076390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.076390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.076444 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.076692 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:19.076693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:19.076729 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:01:19.076790 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:01:19.076903 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":5}
I20260620 23:01:19.077042 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.068799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.1.166.61:36575)
0620 23:01:19.068966 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.068971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.068986 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:01:19.069003 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.069005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.069015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.069070 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.069443 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.070331 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.071535 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.071906 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.071967 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.074149 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.074159 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.074160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.074166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.075974 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.076273 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.076276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.076276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.076326 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.076492 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.076493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.076493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.076634 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.076704 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.076818 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.076952 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260620 23:01:19.077633 24073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:19.077972 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.077861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:60974)
0620 23:01:19.077902 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.077946 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 23:01:19.078099 24021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
I20260620 23:01:19.078526 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.078441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:60988)
0620 23:01:19.078478 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.078502 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 23:01:19.078627 24073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46467: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab.
Entry for principal kudu/127.1.166.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab.
I20260620 23:01:19.090291 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.60:46467
--webserver_interface=127.1.166.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab
--principal=kudu/127.1.166.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:41427,127.1.166.61:36575,127.1.166.60:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:01:19.127907 24073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } attempt: 1
I20260620 23:01:19.128569 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.128382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:60990)
0620 23:01:19.128473 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.128524 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 23:01:19.128577 23998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } attempt: 3
I20260620 23:01:19.128927 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.128825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:60996)
0620 23:01:19.128859 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.128881 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 23:01:19.129026 24073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46467: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996478, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
I20260620 23:01:19.136822 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.128863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58208)
0620 23:01:19.128910 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.128924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:19.128971 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:19.129199 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:19.129200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:19.129362 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:19.129440 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.129442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.130520 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.130523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.131173 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.131175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.131830 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.131832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.131932 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.132456 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:19.132462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:19.133950 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:19.135743 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.135744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.135745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.135907 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.136118 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.136119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.136120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.136188 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.136472 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:19.136473 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:19.136526 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:01:19.136636 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:01:19.136750 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 23:01:19.136879 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.128931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.1.166.61:36575)
0620 23:01:19.129048 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.129050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.129060 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:19.129087 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.129088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.129100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.129164 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.129507 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.130275 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.131321 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.131691 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.131794 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.134057 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.134059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.134059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.134067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.135644 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.135972 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.135973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.135974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.136031 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.136248 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.136249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.136249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.136390 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.136455 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.136702 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.136788 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:01:19.137253 23998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:19.137650 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.137492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32772)
0620 23:01:19.137597 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.137627 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:01:19.137955 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.137885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32784)
0620 23:01:19.137917 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.137939 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 23:01:19.138024 23998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46467: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
I20260620 23:01:19.169193 23998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } attempt: 1
I20260620 23:01:19.169713 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.169581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32792)
0620 23:01:19.169642 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.169681 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 23:01:19.170161 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.170008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32796)
0620 23:01:19.170101 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.170133 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 23:01:19.170281 23998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46467: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
W20260620 23:01:19.184989 24080 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:01:19.185225 24080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:19.185248 24080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:19.185277 24080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:19.186707 24080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:19.186766 24080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:19.186787 24080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:19.186803 24080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:19.186834 24080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:19.188441 24080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:41427,127.1.166.61:36575,127.1.166.60:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab
--principal=kudu/127.1.166.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.60:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.166.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:19.188694 24080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:19.188915 24080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:19.194156 24086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:19.194280 24085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:19.194525 24080 server_base.cc:1034] running on GCE node
W20260620 23:01:19.194597 24088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:19.194903 24080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:19.195272 24080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:19.196466 24080 hybrid_clock.cc:648] HybridClock initialized: now 1781996479196432 us; error 65 us; skew 500 ppm
I20260620 23:01:19.197214 24073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } attempt: 2
I20260620 23:01:19.197742 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.197596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32808)
0620 23:01:19.197664 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.197704 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 23:01:19.198172 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.197998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32812)
0620 23:01:19.198032 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.198113 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 23:01:19.198377 24073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46467: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46467: connect: Connection refused (error 111)
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:19.200012 24080 init.cc:377] Logged in from keytab as kudu/127.1.166.60@KRBTEST.COM (short username kudu)
I20260620 23:01:19.200726 24080 webserver.cc:460] Webserver started at http://127.1.166.60:44003/ using document root <none> and password file <none>
I20260620 23:01:19.200980 24080 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:19.201033 24080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:19.201159 24080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:19.202225 24080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/data/instance:
uuid: "7ad8935382994cd29829b5f4e1318751"
format_stamp: "Formatted at 2026-06-20 23:01:19 on dist-test-slave-rdl9"
server_key: "7a975b7fcbdec75b91410783304954c7"
server_key_iv: "9cd5d30355bd2c79132096c073bbab2e"
server_key_version: "encryptionkey@0"
I20260620 23:01:19.202595 24080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/wal/instance:
uuid: "7ad8935382994cd29829b5f4e1318751"
format_stamp: "Formatted at 2026-06-20 23:01:19 on dist-test-slave-rdl9"
server_key: "7a975b7fcbdec75b91410783304954c7"
server_key_iv: "9cd5d30355bd2c79132096c073bbab2e"
server_key_version: "encryptionkey@0"
I20260620 23:01:19.204063 24080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:19.204952 24095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:19.205164 24080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:19.205257 24080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/wal
uuid: "7ad8935382994cd29829b5f4e1318751"
format_stamp: "Formatted at 2026-06-20 23:01:19 on dist-test-slave-rdl9"
server_key: "7a975b7fcbdec75b91410783304954c7"
server_key_iv: "9cd5d30355bd2c79132096c073bbab2e"
server_key_version: "encryptionkey@0"
I20260620 23:01:19.205403 24080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:19.215811 24080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:19.218438 24080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:19.218636 24080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:19.223451 24080 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.60:46467
I20260620 23:01:19.223574 24147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.60:46467 every 8 connection(s)
I20260620 23:01:19.223861 24080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
I20260620 23:01:19.224853 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:19.226926 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 24080
I20260620 23:01:19.226680 24148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:19.227043 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996175763767-1688-0/minicluster-data/master-2/wal/instance
I20260620 23:01:19.227239 1688 external_mini_cluster.cc:1428] Setting key 50bd7155e1f4ed71bb6b2da91a637eed
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
I20260620 23:01:19.237798 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.227610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.1.166.62:41427)
0620 23:01:19.227665 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.227666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.227904 (+ 238us) server_negotiation.cc:407] Connection header received
0620 23:01:19.228563 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.228565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.228577 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.228648 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.229249 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.229768 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.231159 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.231488 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.231603 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.234551 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.234554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.234554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.234564 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.236580 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.237021 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.237022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.237023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.237074 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.237265 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.237266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.237266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.237381 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.237433 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.237608 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.237699 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:01:19.238096 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.227594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:41427 (local address 127.0.0.1:46830)
0620 23:01:19.227781 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.227830 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:19.228411 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:19.228678 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:19.228681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:19.228971 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:19.229137 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.229141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.229869 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.229871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.230984 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.230987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.231666 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.231668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.231824 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.232500 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:19.232511 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:19.234434 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:19.236693 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.236699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.236702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.236939 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.237127 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.237128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.237128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.237206 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.237434 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:19.237438 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:19.237495 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:01:19.237585 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:01:19.237852 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":71,"thread_start_us":90,"threads_started":1}
I20260620 23:01:19.238245 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.227810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.1.166.60:46467)
0620 23:01:19.227942 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.227946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.227975 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:01:19.228421 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.228433 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.228460 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.228548 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.229380 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.230506 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.231747 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.232164 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.232235 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.234746 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.234762 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.234762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.234775 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.236567 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.236951 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.236954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.236955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.237060 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.237234 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.237235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.237235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.237486 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.237741 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.237797 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.238155 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":276,"server-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260620 23:01:19.238857 24148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:19.245409 23998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } attempt: 2
I20260620 23:01:19.246025 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.239126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:36575 (local address 127.0.0.1:58220)
0620 23:01:19.239159 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.239169 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:19.239208 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:19.239349 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:19.239350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:19.239599 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:19.239644 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.239646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.240333 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.240335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.240829 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.240830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.241164 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.241165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.241240 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.241653 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:19.241657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:19.243063 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:19.244932 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.244933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.244933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.245137 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.245354 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.245356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.245356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.245417 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.245706 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:19.245707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:19.245759 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:01:19.245823 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:01:19.245929 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 23:01:19.246025 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.239198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.1.166.61:36575)
0620 23:01:19.239248 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.239250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.239257 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:01:19.239282 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.239283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.239293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.239339 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.239743 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.240265 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.240898 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.241105 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.241169 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.243164 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.243165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.243166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.243174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.244837 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.245207 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.245208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.245209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.245270 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.245475 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.245476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.245476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.245619 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.245676 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.245850 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.245929 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:19.246380 24148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:19.246658 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.239152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.1.166.62:41427)
0620 23:01:19.239193 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.239195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.239241 (+ 46us) server_negotiation.cc:407] Connection header received
0620 23:01:19.239292 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.239293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.239303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.239345 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.239812 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.240295 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.240994 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.241179 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.241237 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.243652 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.243653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.243653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.243661 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.244975 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.245323 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.245324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.245325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.245395 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.245606 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.245607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.245607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.245737 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.245888 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.246491 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.246575 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996478, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
I20260620 23:01:19.254300 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.245720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.1.166.60:46467)
0620 23:01:19.245764 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.245766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.245773 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:01:19.245792 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.245792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.245801 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.246148 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.246181 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.247174 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.248106 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.248403 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.248458 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.251621 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.251622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.251623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.251636 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.253115 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.253500 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.253500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.253501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.253557 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.253771 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.253772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.253772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.253915 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.253980 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.254128 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.254200 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:01:19.254310 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.245661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32824)
0620 23:01:19.245710 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.245720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:19.245751 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:19.245886 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:19.245887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:19.246033 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:19.246080 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.246082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.247252 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.247253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.248029 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.248032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.248508 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.248509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.248571 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.249001 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:19.249005 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:19.251364 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:19.253222 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.253224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.253225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.253442 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.253628 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.253628 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.253629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.253698 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.253983 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:19.253985 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:19.254036 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:01:19.254098 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:01:19.254202 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 23:01:19.255995 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.247570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.1.166.61:36575)
0620 23:01:19.247601 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.247603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.247685 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:01:19.247727 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.247728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.247738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.247788 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.248302 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.248957 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.249677 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.250060 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.250379 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.252610 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.252611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.252611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.252621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.254217 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.255253 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.255254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.255254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.255309 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.255507 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.255507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.255508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.255638 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.255796 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.255832 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.255904 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 23:01:19.257323 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:19.257359 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.246636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32832)
0620 23:01:19.246696 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.246707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:19.246770 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:19.247390 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:19.247391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:19.247515 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:19.247540 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.247541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.248240 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.248241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.248861 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.248863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.249293 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.249293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.249336 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.250466 (+ 1130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:19.250471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:19.252637 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:19.256201 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.256202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.256202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.256385 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.256563 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.256564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.256564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.256618 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.256887 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:19.256888 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:19.256986 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 23:01:19.257051 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:01:19.257257 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:01:19.257306 23998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33: Bootstrap starting.
I20260620 23:01:19.257455 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.246669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.1.166.60:46467)
0620 23:01:19.247298 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.247300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.247305 (+ 5us) server_negotiation.cc:407] Connection header received
0620 23:01:19.247321 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.247322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.247330 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.247388 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.247600 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.248170 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.248937 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.249197 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.249253 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.252718 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.252719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.252719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.252726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.255997 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.256452 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.256453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.256453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.256516 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.256665 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.256666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.256666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.256825 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.256895 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.257080 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.257313 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":607,"thread_start_us":98,"threads_started":1}
I20260620 23:01:19.258595 23998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:19.258991 23998 log.cc:824] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:19.258773 23972 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1888da909a6f4fb995bb31718c636f33"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46844
I20260620 23:01:19.260211 23998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33: No bootstrap required, opened a new log
I20260620 23:01:19.260895 24148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751: Bootstrap starting.
I20260620 23:01:19.261677 24148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:19.261718 23998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } }
I20260620 23:01:19.261897 23998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1888da909a6f4fb995bb31718c636f33, State: Initialized, Role: FOLLOWER
I20260620 23:01:19.261967 24148 log.cc:824] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:19.261992 23998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } }
I20260620 23:01:19.262358 24156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.262435 23998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:19.262462 24156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:19.262598 23972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:01:19.262660 23972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:19.262897 24148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751: No bootstrap required, opened a new log
I20260620 23:01:19.263602 23972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } }
I20260620 23:01:19.263885 23972 leader_election.cc:288] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [CANDIDATE]: Term 1 election: Requested vote from peers ba18fdbe5b964e129736da755205d4b2 (127.1.166.61:36575), 7ad8935382994cd29829b5f4e1318751 (127.1.166.60:46467)
I20260620 23:01:19.265041 24148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } }
I20260620 23:01:19.265218 24148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad8935382994cd29829b5f4e1318751, State: Initialized, Role: FOLLOWER
I20260620 23:01:19.265312 24148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } }
I20260620 23:01:19.265610 24148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:19.265664 24122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1888da909a6f4fb995bb31718c636f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ad8935382994cd29829b5f4e1318751"
I20260620 23:01:19.265793 24122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:01:19.266263 24167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:19.266422 24167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:19.266832 24122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1888da909a6f4fb995bb31718c636f33 in term 1.
I20260620 23:01:19.267295 23946 leader_election.cc:302] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1888da909a6f4fb995bb31718c636f33, 7ad8935382994cd29829b5f4e1318751; no voters:
I20260620 23:01:19.267490 24156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:19.267654 24156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [term 1 LEADER]: Becoming Leader. State: Replica: 1888da909a6f4fb995bb31718c636f33, State: Running, Role: LEADER
I20260620 23:01:19.267763 24156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } }
I20260620 23:01:19.268265 24156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1888da909a6f4fb995bb31718c636f33. Latest consensus state: current_term: 1 leader_uuid: "1888da909a6f4fb995bb31718c636f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.268378 24156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:19.265614 23930 master_runner.cc:186] Error getting master registration for 127.1.166.61:36575: OK, instance_id { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" instance_seqno: 1781996479052297 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:01:19.269234 23930 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:01:19.268141 24184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:19.269726 24184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:19.268275 24173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.269923 24173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:19.270181 24188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:19.271904 24080 master_runner.cc:186] Error getting master registration for 127.1.166.61:36575: OK, instance_id { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" instance_seqno: 1781996479052297 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:01:19.274299 24080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:01:19.270473 24188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:19.283388 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.269195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.1.166.62:41427)
0620 23:01:19.269237 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.269240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.274422 (+ 5182us) server_negotiation.cc:407] Connection header received
0620 23:01:19.274476 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.274477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.274491 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.274555 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.275364 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.276080 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.277158 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.277476 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.277519 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.279563 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.279565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.279565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.279574 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.281223 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.281932 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.281933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.281934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.282022 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.282486 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.282488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.282488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.282666 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.282765 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.283168 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.283257 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:01:19.287134 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.268744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.1.166.61:36575)
0620 23:01:19.268799 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.268801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.270644 (+ 1843us) server_negotiation.cc:407] Connection header received
0620 23:01:19.270814 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.270817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.270835 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.270897 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.272704 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.273607 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.276259 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.276578 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.276628 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.283599 (+ 6971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.283602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.283602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.283612 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.285406 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.286010 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.286012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.286012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.286077 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.286374 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.286375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.286375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.286524 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.286622 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.286931 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.287027 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:01:19.289520 24073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } attempt: 3
I20260620 23:01:19.293099 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.268999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.1.166.60:46467)
0620 23:01:19.269476 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.269479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.270969 (+ 1490us) server_negotiation.cc:407] Connection header received
0620 23:01:19.271057 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.271058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.271099 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.271174 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.280664 (+ 9490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.281515 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.284560 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.285775 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.285819 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.287310 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.287311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.287312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.287320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.289008 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.289416 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.289417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.289418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.289515 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.289736 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.289737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.289737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.289887 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.290002 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.292893 (+ 2891us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.292997 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260620 23:01:19.295331 24122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [term 1 FOLLOWER]: Refusing update from remote peer 1888da909a6f4fb995bb31718c636f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Jun 20 23:01:19 dist-test-slave-rdl9 krb5kdc[23917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996479, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
I20260620 23:01:19.295745 24185 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:19.298051 24173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1888da909a6f4fb995bb31718c636f33. Latest consensus state: current_term: 1 leader_uuid: "1888da909a6f4fb995bb31718c636f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.298159 24173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:19.301412 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.289807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46467 (local address 127.0.0.1:32844)
0620 23:01:19.289895 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:19.289904 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:19.289941 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:19.290191 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:19.290192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:19.290399 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:19.290461 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.290463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.291842 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.291843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.292737 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:19.292740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.293246 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.293248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:19.293310 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.293921 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:19.293926 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:19.295967 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:19.299758 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.299760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.299761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.300018 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.300411 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:19.300411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:19.300412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:19.300486 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:19.301053 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:19.301054 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:19.301115 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 23:01:19.301187 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 23:01:19.301305 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:01:19.301659 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:19.290045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32844 (local address 127.1.166.60:46467)
0620 23:01:19.290086 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.290088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.290093 (+ 5us) server_negotiation.cc:407] Connection header received
0620 23:01:19.290110 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.290111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.290121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.290180 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.290780 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.291718 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.292816 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:19.293161 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:19.293232 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:19.296065 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:19.296066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:19.296067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:19.296074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:19.299548 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.300098 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.300099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.300100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.300245 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:19.300548 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:19.300549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:19.300550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:19.300812 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:19.300910 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:19.301242 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 23:01:19.301379 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:01:19.303625 24073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2: Bootstrap starting.
I20260620 23:01:19.303865 24185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1888da909a6f4fb995bb31718c636f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.303974 24185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:19.304373 24073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:19.304478 24173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1888da909a6f4fb995bb31718c636f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.304561 24173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:19.304651 24073 log.cc:824] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:19.305521 24192 mvcc.cc:205] Tried to move back new op lower bound from 7299057579168964608 to 7299057579081609216. Current Snapshot: MvccSnapshot[applied={T|T < 7299057579168964608}]
I20260620 23:01:19.305711 24193 mvcc.cc:205] Tried to move back new op lower bound from 7299057579168964608 to 7299057579081609216. Current Snapshot: MvccSnapshot[applied={T|T < 7299057579168964608}]
I20260620 23:01:19.306694 24188 catalog_manager.cc:1347] Generated new cluster ID: bdafa94bcf554199afb197aff63bae2c
I20260620 23:01:19.306746 24188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:19.307459 24073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2: No bootstrap required, opened a new log
I20260620 23:01:19.310103 24073 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } }
I20260620 23:01:19.310287 24073 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba18fdbe5b964e129736da755205d4b2, State: Initialized, Role: FOLLOWER
I20260620 23:01:19.310441 24073 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } }
I20260620 23:01:19.310804 24073 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:19.310927 24046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:19.311942 24046 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [term 1 FOLLOWER]: Refusing update from remote peer 1888da909a6f4fb995bb31718c636f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:19.312196 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1888da909a6f4fb995bb31718c636f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ba18fdbe5b964e129736da755205d4b2"
I20260620 23:01:19.312280 24156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:19.313004 24047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1888da909a6f4fb995bb31718c636f33 in term 1.
I20260620 23:01:19.317925 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1888da909a6f4fb995bb31718c636f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.318058 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:19.318107 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1888da909a6f4fb995bb31718c636f33. Latest consensus state: current_term: 1 leader_uuid: "1888da909a6f4fb995bb31718c636f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.318149 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:19.318171 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1888da909a6f4fb995bb31718c636f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.318207 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:19.319631 24211 mvcc.cc:205] Tried to move back new op lower bound from 7299057579168964608 to 7299057579081609216. Current Snapshot: MvccSnapshot[applied={T|T < 7299057579168964608}]
W20260620 23:01:19.320418 24209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:19.320485 24209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:19.326582 24156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1888da909a6f4fb995bb31718c636f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1888da909a6f4fb995bb31718c636f33" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 41427 } } peers { permanent_uuid: "ba18fdbe5b964e129736da755205d4b2" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 36575 } } peers { permanent_uuid: "7ad8935382994cd29829b5f4e1318751" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46467 } } }
I20260620 23:01:19.326705 24156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:19.330444 24188 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:19.331313 24188 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:19.337105 24188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1888da909a6f4fb995bb31718c636f33: Generated new TSK 0
I20260620 23:01:19.337423 24188 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:01:19.372180 24002 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.370752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.1.166.62:41427)
0620 23:01:19.370800 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.370802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.371473 (+ 671us) server_negotiation.cc:407] Connection header received
0620 23:01:19.371579 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.371580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.371592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.371621 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.372101 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46870: BlockingRecv error: recv got EOF from 127.0.0.1:46870 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 23:01:19.373005 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.371181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.1.166.61:36575)
0620 23:01:19.371222 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.371224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.372453 (+ 1229us) server_negotiation.cc:407] Connection header received
0620 23:01:19.372549 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.372550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.372562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.372590 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.372937 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58240: BlockingRecv error: recv got EOF from 127.0.0.1:58240 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 23:01:19.374081 24152 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.373378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.1.166.60:46467)
0620 23:01:19.373432 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.373434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.373558 (+ 124us) server_negotiation.cc:407] Connection header received
0620 23:01:19.373648 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.373649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.373660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.373689 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.374015 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32860: BlockingRecv error: recv got EOF from 127.0.0.1:32860 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 23:01:19.376708 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.375835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.1.166.61:36575)
0620 23:01:19.375877 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.375879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.376101 (+ 222us) server_negotiation.cc:407] Connection header received
0620 23:01:19.376212 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.376213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.376234 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.376264 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.376652 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58250: BlockingRecv error: recv got EOF from 127.0.0.1:58250 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:01:19.380148 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.379280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.1.166.61:36575)
0620 23:01:19.379318 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.379320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.379586 (+ 266us) server_negotiation.cc:407] Connection header received
0620 23:01:19.379686 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.379687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.379699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.379728 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.380092 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58266: BlockingRecv error: recv got EOF from 127.0.0.1:58266 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:01:19.384500 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.383545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.1.166.61:36575)
0620 23:01:19.383606 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.383608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.383891 (+ 283us) server_negotiation.cc:407] Connection header received
0620 23:01:19.383999 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.384001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.384013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.384044 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.384443 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58270: BlockingRecv error: recv got EOF from 127.0.0.1:58270 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 23:01:19.391222 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.390138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.1.166.61:36575)
0620 23:01:19.390192 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.390194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.390558 (+ 364us) server_negotiation.cc:407] Connection header received
0620 23:01:19.390671 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.390673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.390686 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.390716 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.391140 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58278: BlockingRecv error: recv got EOF from 127.0.0.1:58278 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 23:01:19.401770 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.400700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.1.166.61:36575)
0620 23:01:19.400764 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.400766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.401097 (+ 331us) server_negotiation.cc:407] Connection header received
0620 23:01:19.401204 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.401205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.401217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.401253 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.401690 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58280: BlockingRecv error: recv got EOF from 127.0.0.1:58280 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 23:01:19.420982 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.419376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.1.166.61:36575)
0620 23:01:19.419427 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.419429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.420085 (+ 656us) server_negotiation.cc:407] Connection header received
0620 23:01:19.420222 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.420224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.420241 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.420277 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.420896 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58288: BlockingRecv error: recv got EOF from 127.0.0.1:58288 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 23:01:19.455276 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.454558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.1.166.61:36575)
0620 23:01:19.454610 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.454612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.454737 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:01:19.454802 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.454803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.454816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.454867 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.455195 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58304: BlockingRecv error: recv got EOF from 127.0.0.1:58304 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 23:01:19.521476 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.520562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.1.166.61:36575)
0620 23:01:19.520656 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.520658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.520712 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:01:19.520759 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.520760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.520771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.520831 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.521412 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58318: BlockingRecv error: recv got EOF from 127.0.0.1:58318 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 23:01:19.651279 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.650683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.1.166.61:36575)
0620 23:01:19.650783 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.650785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.650814 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:01:19.650842 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.650843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.650855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.650897 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.651229 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58332: BlockingRecv error: recv got EOF from 127.0.0.1:58332 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 23:01:19.909410 24077 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:19.908767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58334 (local address 127.1.166.61:36575)
0620 23:01:19.908836 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:19.908838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:19.908849 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:19.908884 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:19.908885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:19.908900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:19.908958 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:19.909343 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58334: BlockingRecv error: recv got EOF from 127.0.0.1:58334 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 23:01:20.270305 24184 catalog_manager.cc:1259] Loaded cluster ID: bdafa94bcf554199afb197aff63bae2c
I20260620 23:01:20.270417 24184 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751: loading cluster ID for follower catalog manager: success
I20260620 23:01:20.271174 24184 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751: acquiring CA information for follower catalog manager: success
I20260620 23:01:20.271438 24184 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ad8935382994cd29829b5f4e1318751: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:01:20.320971 24209 catalog_manager.cc:1259] Loaded cluster ID: bdafa94bcf554199afb197aff63bae2c
I20260620 23:01:20.321059 24209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2: loading cluster ID for follower catalog manager: success
I20260620 23:01:20.321918 24209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2: acquiring CA information for follower catalog manager: success
I20260620 23:01:20.322263 24209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba18fdbe5b964e129736da755205d4b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:01:20.425796 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:20.423096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.1.166.61:36575)
0620 23:01:20.423258 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:20.423260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:20.423271 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:20.423304 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:20.423305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:20.423384 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:20.423476 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:20.423671 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.424176 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.424947 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.425144 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.425209 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:20.425413 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:20.425526 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:20.425596 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:20.425681 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 23:01:20.425705 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260620 23:01:20.430914 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:20.427945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.1.166.60:46467)
0620 23:01:20.428054 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:20.428055 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:20.428063 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:20.428103 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:20.428104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:20.428258 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:20.428385 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:20.428618 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.429112 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.429912 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.430107 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.430171 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:20.430433 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:20.430546 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:20.430615 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:20.430809 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:01:20.430831 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
I20260620 23:01:20.436499 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:20.433340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.1.166.61:36575)
0620 23:01:20.433408 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:20.433410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:20.433498 (+ 88us) server_negotiation.cc:407] Connection header received
0620 23:01:20.433606 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:20.433607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:20.433618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:20.433645 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:20.433858 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.434714 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.435759 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.435983 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.436043 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:20.436213 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:20.436260 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:20.436316 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:20.436387 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:01:20.436414 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 23:01:20.436739 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:20.433418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.1.166.60:46467)
0620 23:01:20.433490 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:20.433493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:20.433575 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:01:20.433809 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:20.433810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:20.433820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:20.433871 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:20.433977 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.434518 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.435632 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.435946 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.436021 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:20.436176 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:20.436217 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:20.436268 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:20.436652 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 23:01:20.436686 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:01:20.437177 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:20.433145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.1.166.62:41427)
0620 23:01:20.433291 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:20.433292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:20.433302 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:20.433449 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:20.433450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:20.433536 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:20.433645 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:20.433820 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.434756 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.435661 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:20.435999 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:20.436082 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:20.436313 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:20.436454 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:20.437003 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:20.437021 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 23:01:20.437054 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260620 23:01:20.439962 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 23930
I20260620 23:01:20.445744 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 24005
I20260620 23:01:20.450511 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 24080
2026-06-20T23:01:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2797 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:20 dist-test-slave-rdl9 krb5kdc[24297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:20 dist-test-slave-rdl9 krb5kdc[24297](info): set up 2 sockets
Jun 20 23:01:20 dist-test-slave-rdl9 krb5kdc[24297](info): commencing operation
krb5kdc: starting...
W20260620 23:01:21.426484 1688 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/test-user.keytab.
Jun 20 23:01:21 dist-test-slave-rdl9 krb5kdc[24297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
Entry for principal kudu/127.1.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab.
I20260620 23:01:21.547657 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:39991
--webserver_interface=127.1.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:39991,127.1.166.61:46883,127.1.166.60:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:21.635497 24310 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:01:21.635736 24310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:21.635769 24310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:21.635799 24310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:21.637006 24310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:21.637050 24310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:21.637069 24310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:21.637085 24310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:21.637101 24310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:21.638672 24310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:39991,127.1.166.61:46883,127.1.166.60:46805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.62.keytab
--principal=kudu/127.1.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.62:39991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:21.638932 24310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:21.639145 24310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:21.643937 24315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:21.643947 24318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:21.643952 24316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:21.644318 24310 server_base.cc:1034] running on GCE node
I20260620 23:01:21.644585 24310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:21.644904 24310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:21.646085 24310 hybrid_clock.cc:648] HybridClock initialized: now 1781996481646068 us; error 34 us; skew 500 ppm
Jun 20 23:01:21 dist-test-slave-rdl9 krb5kdc[24297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:21.651970 24310 init.cc:377] Logged in from keytab as kudu/127.1.166.62@KRBTEST.COM (short username kudu)
I20260620 23:01:21.652598 24310 webserver.cc:460] Webserver started at http://127.1.166.62:37597/ using document root <none> and password file <none>
I20260620 23:01:21.653045 24310 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:21.653232 24310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:21.654096 24310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:21.655812 24310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/data/instance:
uuid: "1a4a6d6926de4f9796706b98e08deefa"
format_stamp: "Formatted at 2026-06-20 23:01:21 on dist-test-slave-rdl9"
server_key: "f57879ebba7dc23f0380222f7fd00eba"
server_key_iv: "ab3b289fb23797a901acd602043a6dca"
server_key_version: "encryptionkey@0"
I20260620 23:01:21.656311 24310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/wal/instance:
uuid: "1a4a6d6926de4f9796706b98e08deefa"
format_stamp: "Formatted at 2026-06-20 23:01:21 on dist-test-slave-rdl9"
server_key: "f57879ebba7dc23f0380222f7fd00eba"
server_key_iv: "ab3b289fb23797a901acd602043a6dca"
server_key_version: "encryptionkey@0"
I20260620 23:01:21.658639 24310 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 23:01:21.659813 24325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:21.660075 24310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:21.660180 24310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/wal
uuid: "1a4a6d6926de4f9796706b98e08deefa"
format_stamp: "Formatted at 2026-06-20 23:01:21 on dist-test-slave-rdl9"
server_key: "f57879ebba7dc23f0380222f7fd00eba"
server_key_iv: "ab3b289fb23797a901acd602043a6dca"
server_key_version: "encryptionkey@0"
I20260620 23:01:21.660271 24310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:21.674921 24310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:21.677495 24310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:21.677775 24310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:21.682890 24310 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.62:39991
I20260620 23:01:21.682924 24377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.62:39991 every 8 connection(s)
I20260620 23:01:21.683291 24310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:21.684278 24378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:21.685894 24378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:01:21 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:21.693298 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 24310
I20260620 23:01:21.693408 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-0/wal/instance
I20260620 23:01:21.693655 1688 external_mini_cluster.cc:1428] Setting key df5253c19057e81529aa080555fa2490
I20260620 23:01:21.698086 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.686831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.1.166.62:39991)
0620 23:01:21.686952 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:21.686956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:21.686983 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:01:21.687502 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:21.687505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:21.687527 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:21.687611 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:21.688192 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.689067 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.690096 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.690662 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.690810 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:21.694065 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:21.694074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:21.694074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:21.694083 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:21.695862 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:21.696352 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:21.696356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:21.696356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:21.696456 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:21.696782 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:21.696784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:21.696785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:21.697200 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:21.697354 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:21.697675 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:01:21.697835 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":92,"thread_start_us":38,"threads_started":1}
I20260620 23:01:21.698093 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.686668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39991 (local address 127.0.0.1:42676)
0620 23:01:21.686825 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.686862 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:21.687340 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:21.687633 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:21.687636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:21.687898 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:21.688078 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:21.688082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.689182 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.689184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:21.689938 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:21.689940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.690879 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.690881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:21.690972 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:21.691676 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:21.691688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:21.693940 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:21.695969 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:21.695977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:21.695980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:21.696243 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:21.696536 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:21.696537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:21.696537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:21.696644 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:21.697340 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:21.697348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:21.697470 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 23:01:21.697597 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:01:21.697915 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":129,"thread_start_us":72,"threads_started":1}
I20260620 23:01:21.698885 24378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:21.699247 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.699109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52210)
0620 23:01:21.699160 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.699220 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 23:01:21.699435 24326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
I20260620 23:01:21.699978 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.699869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52224)
0620 23:01:21.699922 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.699954 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:01:21.700201 24378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:46883: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.166.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab.
Entry for principal kudu/127.1.166.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab.
I20260620 23:01:21.720570 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.61:46883
--webserver_interface=127.1.166.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab
--principal=kudu/127.1.166.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:39991,127.1.166.61:46883,127.1.166.60:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:01:21.749543 24378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } attempt: 1
I20260620 23:01:21.750391 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.750116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52232)
0620 23:01:21.750248 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.750300 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 23:01:21.750839 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.750708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52248)
0620 23:01:21.750779 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.750814 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:01:21.750972 24378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:46883: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
W20260620 23:01:21.811976 24385 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:01:21.812294 24385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:21.812314 24385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:21.812341 24385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:21.813936 24385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:21.813984 24385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:21.813998 24385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:21.814010 24385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:21.814021 24385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:21.817130 24385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:39991,127.1.166.61:46883,127.1.166.60:46805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.61.keytab
--principal=kudu/127.1.166.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.61:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.166.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:21.817437 24385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:21.817849 24385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:01:21.819209 24378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } attempt: 2
I20260620 23:01:21.819777 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.819630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52254)
0620 23:01:21.819704 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.819744 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:01:21.820366 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.820155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52258)
0620 23:01:21.820267 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.820320 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 23:01:21.820539 24378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.61:46883: Network error: Client connection negotiation failed: client connection to 127.1.166.61:46883: connect: Connection refused (error 111)
W20260620 23:01:21.823410 24391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:21.823407 24393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:21.823424 24390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:21.823678 24385 server_base.cc:1034] running on GCE node
I20260620 23:01:21.823977 24385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:21.824319 24385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:21.825506 24385 hybrid_clock.cc:648] HybridClock initialized: now 1781996481825489 us; error 29 us; skew 500 ppm
Jun 20 23:01:21 dist-test-slave-rdl9 krb5kdc[24297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:21.828315 24385 init.cc:377] Logged in from keytab as kudu/127.1.166.61@KRBTEST.COM (short username kudu)
I20260620 23:01:21.829126 24385 webserver.cc:460] Webserver started at http://127.1.166.61:44095/ using document root <none> and password file <none>
I20260620 23:01:21.829433 24385 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:21.829504 24385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:21.829653 24385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:21.831192 24385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/data/instance:
uuid: "35495892363c499cb25bb3a42d71faeb"
format_stamp: "Formatted at 2026-06-20 23:01:21 on dist-test-slave-rdl9"
server_key: "80417c9e8c2542a79b047b9983498995"
server_key_iv: "5f5e185a48176642b52955196d39993e"
server_key_version: "encryptionkey@0"
I20260620 23:01:21.831626 24385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/wal/instance:
uuid: "35495892363c499cb25bb3a42d71faeb"
format_stamp: "Formatted at 2026-06-20 23:01:21 on dist-test-slave-rdl9"
server_key: "80417c9e8c2542a79b047b9983498995"
server_key_iv: "5f5e185a48176642b52955196d39993e"
server_key_version: "encryptionkey@0"
I20260620 23:01:21.833031 24385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:21.834033 24400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:21.834263 24385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:21.834393 24385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/wal
uuid: "35495892363c499cb25bb3a42d71faeb"
format_stamp: "Formatted at 2026-06-20 23:01:21 on dist-test-slave-rdl9"
server_key: "80417c9e8c2542a79b047b9983498995"
server_key_iv: "5f5e185a48176642b52955196d39993e"
server_key_version: "encryptionkey@0"
I20260620 23:01:21.834496 24385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:21.872198 24385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:21.874928 24385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:21.875097 24385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:21.879149 24385 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.61:46883
I20260620 23:01:21.879194 24452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.61:46883 every 8 connection(s)
I20260620 23:01:21.879554 24385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/data/info.pb
I20260620 23:01:21.880550 24453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:21.882086 24453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:21.887043 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 24385
I20260620 23:01:21.887152 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-1/wal/instance
I20260620 23:01:21.887379 1688 external_mini_cluster.cc:1428] Setting key aa6b56b4a60f688db12e51b3a963a3bf
Jun 20 23:01:21 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:21.891386 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.882871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.1.166.62:39991)
0620 23:01:21.882946 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:21.882948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:21.883106 (+ 158us) server_negotiation.cc:407] Connection header received
0620 23:01:21.883598 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:21.883600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:21.883615 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:21.883664 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:21.884269 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.884767 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.885736 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.885948 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.886005 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:21.888334 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:21.888335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:21.888336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:21.888344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:21.889710 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:21.890131 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:21.890132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:21.890132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:21.890192 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:21.890654 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:21.890655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:21.890655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:21.890822 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:21.890895 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:21.891199 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:01:21.891288 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:01:21.891494 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.882826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39991 (local address 127.0.0.1:42688)
0620 23:01:21.882995 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.883030 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:21.883505 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:21.883740 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:21.883743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:21.883989 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:21.884184 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:21.884188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.884852 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.884853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:21.885654 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:21.885656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.886031 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.886032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:21.886164 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:21.886670 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:21.886679 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:21.888222 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:21.889812 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:21.889820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:21.889822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:21.890061 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:21.890287 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:21.890288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:21.890288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:21.890453 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:21.890906 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:21.890913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:21.891058 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 23:01:21.891161 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:01:21.891305 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260620 23:01:21.892441 24453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.166.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.166.60@KRBTEST.COM" created.
Jun 20 23:01:21 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
I20260620 23:01:21.901714 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.892743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52268)
0620 23:01:21.892799 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.892811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:21.892864 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:21.893118 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:21.893119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:21.893284 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:21.893355 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:21.893357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.894248 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.894249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:21.895087 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:21.895090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.895875 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.895877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:21.895957 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:21.896479 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:21.896484 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:21.897941 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:21.900398 (+ 2457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:21.900401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:21.900402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:21.900714 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:21.901014 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:21.901015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:21.901016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:21.901081 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:21.901375 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:21.901376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:21.901427 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:01:21.901495 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:01:21.901613 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 23:01:21.901841 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.892809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.1.166.61:46883)
0620 23:01:21.892977 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:21.892982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:21.893001 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:01:21.893023 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:21.893024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:21.893036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:21.893105 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:21.893444 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.894148 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.895192 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.895747 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.895862 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:21.898108 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:21.898124 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:21.898125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:21.898137 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:21.900251 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:21.900855 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:21.900860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:21.900860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:21.900939 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:21.901135 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:21.901136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:21.901136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:21.901314 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:21.901391 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:21.901579 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 23:01:21.901709 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260620 23:01:21.902701 24453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:21.903121 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.902945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36454)
0620 23:01:21.903023 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.903087 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 23:01:21.903280 24403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
I20260620 23:01:21.903702 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.903600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36470)
0620 23:01:21.903646 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.903678 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 23:01:21.903806 24453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46805: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
I20260620 23:01:21.911708 24378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.166.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab.
Entry for principal kudu/127.1.166.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab.
I20260620 23:01:21.914113 1688 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskE9BVnS/build/release/bin/kudu
/tmp/dist-test-taskE9BVnS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.60:46805
--webserver_interface=127.1.166.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab
--principal=kudu/127.1.166.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.166.20:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.166.62:39991,127.1.166.61:46883,127.1.166.60:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:01:21 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
I20260620 23:01:21.920059 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.912160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52276)
0620 23:01:21.912283 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.912302 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:21.912388 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:21.912676 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:21.912678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:21.912866 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:21.913066 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:21.913071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.914143 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.914145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:21.914775 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:21.914778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.915174 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.915175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:21.915244 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:21.915669 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:21.915673 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:21.916871 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:21.918813 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:21.918814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:21.918815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:21.919005 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:21.919193 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:21.919194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:21.919194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:21.919245 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:21.919484 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:21.919485 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:21.919657 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 23:01:21.919749 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:01:21.919922 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:01:21.920053 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.912349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.1.166.61:46883)
0620 23:01:21.912517 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:21.912519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:21.912528 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:21.912559 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:21.912560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:21.912570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:21.912618 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:21.913202 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.914007 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.914867 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:21.915108 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:21.915170 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:21.916968 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:21.916969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:21.916969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:21.916979 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:21.918551 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:21.919079 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:21.919081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:21.919081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:21.919137 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:21.919294 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:21.919295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:21.919295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:21.919423 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:21.919577 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:21.919798 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:01:21.919935 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260620 23:01:21.920629 24378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:21.921118 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.920984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36474)
0620 23:01:21.921045 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.921095 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 23:01:21.921552 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.921451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36480)
0620 23:01:21.921505 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.921532 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 23:01:21.921638 24378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46805: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
I20260620 23:01:21.952828 24378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } attempt: 1
I20260620 23:01:21.953132 24453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } attempt: 1
I20260620 23:01:21.953368 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.953232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36490)
0620 23:01:21.953297 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.953333 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 23:01:21.953609 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.953372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36500)
0620 23:01:21.953529 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.953572 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 23:01:21.954030 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.953889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36514)
0620 23:01:21.953933 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.953988 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 23:01:21.954149 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:21.954010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36524)
0620 23:01:21.954076 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:21.954113 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:01:21.954149 24378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46805: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
W20260620 23:01:21.954413 24453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46805: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
W20260620 23:01:21.999816 24460 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:01:22.000077 24460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:22.000104 24460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:22.000134 24460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:22.001466 24460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:22.001531 24460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:22.001550 24460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:22.001566 24460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:22.001581 24460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:22.003543 24460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.166.20:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.166.62:39991,127.1.166.61:46883,127.1.166.60:46805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/krb5kdc/kudu_127.1.166.60.keytab
--principal=kudu/127.1.166.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.166.60:46805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.166.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:22.003871 24460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:22.004106 24460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:22.009948 24466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:22.009959 24465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:22.009943 24468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:22.010176 24460 server_base.cc:1034] running on GCE node
I20260620 23:01:22.010454 24460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:22.010866 24460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:22.012072 24460 hybrid_clock.cc:648] HybridClock initialized: now 1781996482012036 us; error 100 us; skew 500 ppm
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:22.014711 24460 init.cc:377] Logged in from keytab as kudu/127.1.166.60@KRBTEST.COM (short username kudu)
I20260620 23:01:22.015192 24460 webserver.cc:460] Webserver started at http://127.1.166.60:42313/ using document root <none> and password file <none>
I20260620 23:01:22.015421 24460 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:22.015471 24460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:22.015606 24460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:22.016690 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/data/instance:
uuid: "758b97133be1428e8dffb01b70280a6a"
format_stamp: "Formatted at 2026-06-20 23:01:22 on dist-test-slave-rdl9"
server_key: "d2eeff35541de399b9452b704ccf6b6c"
server_key_iv: "445ed0dbae6ec89d596beb22a22c477e"
server_key_version: "encryptionkey@0"
I20260620 23:01:22.017045 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/wal/instance:
uuid: "758b97133be1428e8dffb01b70280a6a"
format_stamp: "Formatted at 2026-06-20 23:01:22 on dist-test-slave-rdl9"
server_key: "d2eeff35541de399b9452b704ccf6b6c"
server_key_iv: "445ed0dbae6ec89d596beb22a22c477e"
server_key_version: "encryptionkey@0"
I20260620 23:01:22.018436 24460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:01:22.019636 24475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:22.019897 24460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:22.020015 24460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/data,/tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/wal
uuid: "758b97133be1428e8dffb01b70280a6a"
format_stamp: "Formatted at 2026-06-20 23:01:22 on dist-test-slave-rdl9"
server_key: "d2eeff35541de399b9452b704ccf6b6c"
server_key_iv: "445ed0dbae6ec89d596beb22a22c477e"
server_key_version: "encryptionkey@0"
I20260620 23:01:22.020094 24460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:22.022660 24453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } attempt: 2
I20260620 23:01:22.023083 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.022970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36528)
0620 23:01:22.023021 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.023055 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:01:22.023437 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.023352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36532)
0620 23:01:22.023389 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.023420 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 23:01:22.023532 24453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46805: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
I20260620 23:01:22.029313 24378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } attempt: 2
I20260620 23:01:22.029997 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.029810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36544)
0620 23:01:22.029897 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.029955 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:01:22.030465 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.030321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36558)
0620 23:01:22.030421 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.030446 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 23:01:22.030577 24378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.166.60:46805: Network error: Client connection negotiation failed: client connection to 127.1.166.60:46805: connect: Connection refused (error 111)
I20260620 23:01:22.031606 24460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:22.033859 24460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:22.033987 24460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:22.038239 24460 rpc_server.cc:307] RPC server started. Bound to: 127.1.166.60:46805
I20260620 23:01:22.038313 24527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.166.60:46805 every 8 connection(s)
I20260620 23:01:22.038698 24460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/data/info.pb
I20260620 23:01:22.039788 1688 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskE9BVnS/build/release/bin/kudu as pid 24460
I20260620 23:01:22.039856 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:22.039894 1688 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskE9BVnS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996175763767-1688-0/minicluster-data/master-2/wal/instance
I20260620 23:01:22.040099 1688 external_mini_cluster.cc:1428] Setting key f8c4d51f7e37c9b3936f015a66e54146
I20260620 23:01:22.042230 24528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:22.051270 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.040708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.1.166.60:46805)
0620 23:01:22.040904 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.040914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.040952 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:01:22.041445 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.041456 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.041530 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.041696 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.042326 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.043629 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.044721 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.045379 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.045498 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.047892 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.047900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.047901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.047907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.049505 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.050041 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.050044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.050045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.050136 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.050314 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.050315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.050316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.050621 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.050762 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.050873 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.051002 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260620 23:01:22.051437 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.042964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.62:39991 (local address 127.0.0.1:42690)
0620 23:01:22.043137 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.043180 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:22.043264 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:22.043423 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:22.043425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:22.043777 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:22.043842 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.043844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.044739 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.044740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.045580 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.045582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.046017 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.046018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.046078 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.046531 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:22.046540 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:22.048247 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:22.050369 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.050377 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.050380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.050600 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.050790 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.050791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.050791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.050850 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.051082 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:22.051087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:22.051138 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:01:22.051231 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:01:22.051348 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260620 23:01:22.051533 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.043045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.1.166.62:39991)
0620 23:01:22.043112 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.043114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.043243 (+ 129us) server_negotiation.cc:407] Connection header received
0620 23:01:22.043322 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.043323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.043333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.043381 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.043946 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.044644 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.045670 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.045907 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.045977 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.048334 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.048336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.048336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.048344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.050240 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.050675 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.050676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.050676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.050732 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.050906 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.050907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.050908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.051021 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.051087 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.051292 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.051400 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:01:22.052376 24528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.62@KRBTEST.COM
I20260620 23:01:22.060832 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.052693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.61:46883 (local address 127.0.0.1:52280)
0620 23:01:22.052779 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.052794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:22.052851 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:22.053070 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:22.053070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:22.053252 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:22.053298 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.053300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.053969 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.053970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.054563 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.054565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.054936 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.054936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.054998 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.055581 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:22.055587 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:22.057216 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:22.059827 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.059828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.059828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.060022 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.060224 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.060224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.060225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.060279 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.060511 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:22.060513 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:22.060559 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:01:22.060614 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 23:01:22.060724 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:01:22.060832 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.052835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.1.166.61:46883)
0620 23:01:22.052904 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.052906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.052917 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:22.052995 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.052996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.053007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.053064 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.053371 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.053892 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.054642 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.054875 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.054931 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.057330 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.057332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.057332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.057342 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.059704 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.060093 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.060094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.060095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.060163 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.060330 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.060331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.060332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.060454 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.060509 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.060646 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.060724 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:01:22.061386 24528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:22.061486 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.052905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.1.166.62:39991)
0620 23:01:22.052960 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.052962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.052981 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:01:22.053010 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.053012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.053022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.053169 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.053719 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.054194 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.054932 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.055120 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.055175 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.058151 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.058153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.058153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.058162 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.059905 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.060310 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.060311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.060312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.060371 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.060535 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.060536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.060536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.060648 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.060860 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.061309 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.061408 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.60@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.166.61@KRBTEST.COM
I20260620 23:01:22.069064 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.061670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36568)
0620 23:01:22.061760 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.061771 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:22.061820 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:22.061934 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:22.061935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:22.062108 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:22.062150 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.062152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.063113 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.063115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.063834 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.063837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.064397 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.064398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.064447 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.064897 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:22.064902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:22.066333 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:22.068001 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.068002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.068003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.068202 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.068460 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.068460 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.068461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.068517 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.068791 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:22.068791 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:22.068831 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:01:22.068878 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 23:01:22.068971 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:01:22.069157 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.061710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.1.166.60:46805)
0620 23:01:22.061798 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.061800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.061807 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:01:22.061873 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.061874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.061884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.061929 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.062217 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.063035 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.063923 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.064314 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.064402 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.066447 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.066447 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.066448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.066453 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.067884 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.068311 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.068313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.068313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.068384 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.068573 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.068573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.068574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.068725 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.068799 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.068920 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.069049 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":72}
I20260620 23:01:22.070834 24528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a: Bootstrap starting.
I20260620 23:01:22.071134 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.062122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.1.166.61:46883)
0620 23:01:22.062170 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.062172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.062178 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:01:22.062197 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.062198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.062207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.062257 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.062920 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.063580 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.064501 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.064776 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.064843 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.067262 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.067263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.067264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.067273 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.069137 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.069531 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.069532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.069532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.069594 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.070593 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.070594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.070594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.070738 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.070912 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.070959 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.071042 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:01:22.071818 24528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:22.072108 24528 log.cc:824] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:22.072798 1688 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:22.073083 24528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a: No bootstrap required, opened a new log
I20260620 23:01:22.073940 24352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1a4a6d6926de4f9796706b98e08deefa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42692
I20260620 23:01:22.075752 24528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } }
I20260620 23:01:22.075940 24528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758b97133be1428e8dffb01b70280a6a, State: Initialized, Role: FOLLOWER
I20260620 23:01:22.076020 24528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } }
I20260620 23:01:22.076400 24528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:22.076390 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.076711 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:01:22.078325 24545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:22.078665 24545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:22.079012 24460 master_runner.cc:186] Error getting master registration for 127.1.166.62:39991: OK, instance_id { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" instance_seqno: 1781996481681526 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:01:22.082191 24460 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:01:22.114679 24453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } attempt: 3
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.61@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
I20260620 23:01:22.122851 24455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.115058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36582)
0620 23:01:22.115118 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.115130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:22.115171 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:22.115427 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:22.115428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:22.115623 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:22.115678 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.115680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.116570 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.116572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.117414 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.117417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.118020 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.118021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.118079 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.118610 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:22.118615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:22.120305 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:22.121770 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.121771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.121771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.121943 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.122113 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.122113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.122114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.122171 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.122563 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:22.122563 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:22.122612 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:01:22.122670 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:01:22.122770 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:01:22.123202 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.115169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.1.166.60:46805)
0620 23:01:22.115265 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.115267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.115279 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:01:22.115310 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.115311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.115324 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.115413 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.115808 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.116425 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.117579 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.117928 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.118022 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.120416 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.120418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.120418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.120426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.121683 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.122006 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.122007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.122008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.122058 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.122220 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.122221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.122221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.122489 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.122551 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.123072 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.123142 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:01:22.124963 24453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb: Bootstrap starting.
I20260620 23:01:22.125732 24453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:22.126073 24453 log.cc:824] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:22.127058 24453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb: No bootstrap required, opened a new log
I20260620 23:01:22.129170 24453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } }
I20260620 23:01:22.129359 24453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35495892363c499cb25bb3a42d71faeb, State: Initialized, Role: FOLLOWER
I20260620 23:01:22.129447 24453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } }
I20260620 23:01:22.129675 24378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } attempt: 3
I20260620 23:01:22.129851 24453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:22.131321 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.132032 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 23:01:22 dist-test-slave-rdl9 krb5kdc[24297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.166.62@KRBTEST.COM for kudu/127.1.166.60@KRBTEST.COM
I20260620 23:01:22.131923 24385 master_runner.cc:186] Error getting master registration for 127.1.166.62:39991: OK, instance_id { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" instance_seqno: 1781996481681526 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:01:22.134889 24385 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:01:22.131825 24559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:22.134971 24559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:22.138135 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.130022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.166.60:46805 (local address 127.0.0.1:36586)
0620 23:01:22.130084 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:22.130096 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:22.130146 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:22.130299 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:22.130300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:22.130524 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:22.130580 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.130582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.131382 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.131383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.132105 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:22.132108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.132661 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.132662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:22.132718 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.133249 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:22.133255 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:22.134832 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:22.136834 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.136836 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.136837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.137218 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.137453 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:22.137454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:22.137455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:22.137537 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:22.137810 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:22.137811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:22.137859 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:01:22.137920 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:01:22.138033 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 23:01:22.138164 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.130093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.1.166.60:46805)
0620 23:01:22.130138 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.130140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.130149 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:22.130222 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.130223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.130235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.130281 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.130673 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.131291 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.132233 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.132500 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.132557 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.134995 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.134997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.134997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.135005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.136677 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.137319 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.137321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.137321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.137396 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.137583 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.137584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.137584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.137737 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.137811 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.137989 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.138059 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:01:22.140039 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa: Bootstrap starting.
I20260620 23:01:22.140774 24378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:22.141059 24378 log.cc:824] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:22.141866 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa: No bootstrap required, opened a new log
I20260620 23:01:22.143244 24378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } }
I20260620 23:01:22.143400 24378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4a6d6926de4f9796706b98e08deefa, State: Initialized, Role: FOLLOWER
I20260620 23:01:22.143481 24378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } }
I20260620 23:01:22.143814 24378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:22.143857 24352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:01:22.143899 24352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:22.144745 24352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } }
I20260620 23:01:22.145009 24352 leader_election.cc:288] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [CANDIDATE]: Term 1 election: Requested vote from peers 35495892363c499cb25bb3a42d71faeb (127.1.166.61:46883), 758b97133be1428e8dffb01b70280a6a (127.1.166.60:46805)
I20260620 23:01:22.146195 24502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a4a6d6926de4f9796706b98e08deefa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "758b97133be1428e8dffb01b70280a6a"
I20260620 23:01:22.146353 24502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:22.147084 24502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a4a6d6926de4f9796706b98e08deefa in term 1.
I20260620 23:01:22.147325 24328 leader_election.cc:302] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a4a6d6926de4f9796706b98e08deefa, 758b97133be1428e8dffb01b70280a6a; no voters:
I20260620 23:01:22.143756 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.147423 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:22.147485 24562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:22.147524 24562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [term 1 LEADER]: Becoming Leader. State: Replica: 1a4a6d6926de4f9796706b98e08deefa, State: Running, Role: LEADER
I20260620 23:01:22.147593 24562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } }
I20260620 23:01:22.147994 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4a6d6926de4f9796706b98e08deefa. Latest consensus state: current_term: 1 leader_uuid: "1a4a6d6926de4f9796706b98e08deefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.148068 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:22.148389 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a4a6d6926de4f9796706b98e08deefa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35495892363c499cb25bb3a42d71faeb"
I20260620 23:01:22.148520 24427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:22.149185 24427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a4a6d6926de4f9796706b98e08deefa in term 1.
I20260620 23:01:22.150750 24582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:22.151271 24582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:22.153977 24502 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [term 1 FOLLOWER]: Refusing update from remote peer 1a4a6d6926de4f9796706b98e08deefa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:22.154448 24562 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [LEADER]: Connected to new peer: Peer: permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:22.155191 24427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [term 1 FOLLOWER]: Refusing update from remote peer 1a4a6d6926de4f9796706b98e08deefa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:22.156978 24562 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [LEADER]: Connected to new peer: Peer: permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:22.157042 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4a6d6926de4f9796706b98e08deefa. Latest consensus state: current_term: 1 leader_uuid: "1a4a6d6926de4f9796706b98e08deefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.157119 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:22.158289 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4a6d6926de4f9796706b98e08deefa. Latest consensus state: current_term: 1 leader_uuid: "1a4a6d6926de4f9796706b98e08deefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.158412 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:22.160183 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.146158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.1.166.62:39991)
0620 23:01:22.146198 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.146199 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.151662 (+ 5463us) server_negotiation.cc:407] Connection header received
0620 23:01:22.151935 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.151936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.151947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.151975 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.152581 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.153267 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.154234 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.154729 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.154781 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.157357 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.157358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.157359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.157366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.158986 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.159389 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.159390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.159390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.159446 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.159626 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.159627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.159627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.159740 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.159804 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.160006 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.160089 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:01:22.162227 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a4a6d6926de4f9796706b98e08deefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.162317 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:22.162392 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a4a6d6926de4f9796706b98e08deefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.162436 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:22.162618 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a4a6d6926de4f9796706b98e08deefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.162698 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:22.162830 24585 mvcc.cc:205] Tried to move back new op lower bound from 7299057590900006912 to 7299057590877184000. Current Snapshot: MvccSnapshot[applied={T|T < 7299057590900006912}]
I20260620 23:01:22.163010 24548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a4a6d6926de4f9796706b98e08deefa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a4a6d6926de4f9796706b98e08deefa" member_type: VOTER last_known_addr { host: "127.1.166.62" port: 39991 } } peers { permanent_uuid: "35495892363c499cb25bb3a42d71faeb" member_type: VOTER last_known_addr { host: "127.1.166.61" port: 46883 } } peers { permanent_uuid: "758b97133be1428e8dffb01b70280a6a" member_type: VOTER last_known_addr { host: "127.1.166.60" port: 46805 } } }
I20260620 23:01:22.163097 24548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:22.163537 24588 mvcc.cc:205] Tried to move back new op lower bound from 7299057590900006912 to 7299057590877184000. Current Snapshot: MvccSnapshot[applied={T|T < 7299057590900006912 or (T in {7299057590900006912})}]
I20260620 23:01:22.165354 24586 mvcc.cc:205] Tried to move back new op lower bound from 7299057590900006912 to 7299057590877184000. Current Snapshot: MvccSnapshot[applied={T|T < 7299057590900006912}]
I20260620 23:01:22.168298 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.147213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.1.166.60:46805)
0620 23:01:22.150021 (+ 2808us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.150023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.158424 (+ 8401us) server_negotiation.cc:407] Connection header received
0620 23:01:22.158527 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.158528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.158539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.158565 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.159181 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.159848 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.163510 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.163793 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.163862 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.165748 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.165749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.165749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.165756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.167154 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.167526 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.167527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.167528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.167582 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.167776 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.167777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.167777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.167899 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.167965 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.168141 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.168213 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2783}
I20260620 23:01:22.170838 24582 catalog_manager.cc:1347] Generated new cluster ID: 6b00cf91f41a43d1a3d00cfcaa3bef43
I20260620 23:01:22.170899 24582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:22.175045 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:22.150288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.1.166.61:46883)
0620 23:01:22.150329 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.150330 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.159726 (+ 9396us) server_negotiation.cc:407] Connection header received
0620 23:01:22.159820 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.159821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.159832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.159859 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.166641 (+ 6782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.167357 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.170470 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:22.170773 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:22.170816 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:22.172301 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:22.172303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:22.172303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:22.172311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:22.173794 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.174169 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.174170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.174171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.174225 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:22.174466 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:22.174467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:22.174467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:22.174602 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:22.174679 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:22.174869 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 23:01:22.174946 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:01:22.183772 24582 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:22.184345 24582 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:22.207435 24582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a4a6d6926de4f9796706b98e08deefa: Generated new TSK 0
I20260620 23:01:22.207715 24582 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:01:22.233264 24381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.231944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.1.166.62:39991)
0620 23:01:22.231989 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.231991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.232576 (+ 585us) server_negotiation.cc:407] Connection header received
0620 23:01:22.232682 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.232683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.232700 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.232735 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.233190 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42706: BlockingRecv error: recv got EOF from 127.0.0.1:42706 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 23:01:22.234107 24531 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.232283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.1.166.60:46805)
0620 23:01:22.232322 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.232324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.233624 (+ 1300us) server_negotiation.cc:407] Connection header received
0620 23:01:22.233713 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.233714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.233726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.233753 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.234048 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36598: BlockingRecv error: recv got EOF from 127.0.0.1:36598 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 23:01:22.235319 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.234600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.1.166.61:46883)
0620 23:01:22.234666 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.234669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.234844 (+ 175us) server_negotiation.cc:407] Connection header received
0620 23:01:22.234927 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.234928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.234937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.234965 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.235265 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52302: BlockingRecv error: recv got EOF from 127.0.0.1:52302 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 23:01:22.238210 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.237379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.1.166.61:46883)
0620 23:01:22.237429 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.237431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.237657 (+ 226us) server_negotiation.cc:407] Connection header received
0620 23:01:22.237760 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.237761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.237779 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.237806 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.238156 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52306: BlockingRecv error: recv got EOF from 127.0.0.1:52306 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:01:22.241626 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.240793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.1.166.61:46883)
0620 23:01:22.240832 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.240834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.241069 (+ 235us) server_negotiation.cc:407] Connection header received
0620 23:01:22.241167 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.241168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.241180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.241208 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.241569 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52310: BlockingRecv error: recv got EOF from 127.0.0.1:52310 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:01:22.245983 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.245047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.1.166.61:46883)
0620 23:01:22.245098 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.245100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.245380 (+ 280us) server_negotiation.cc:407] Connection header received
0620 23:01:22.245503 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.245505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.245517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.245545 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.245919 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52318: BlockingRecv error: recv got EOF from 127.0.0.1:52318 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 23:01:22.252468 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.251545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.1.166.61:46883)
0620 23:01:22.251593 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.251595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.251861 (+ 266us) server_negotiation.cc:407] Connection header received
0620 23:01:22.251963 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.251964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.251977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.252006 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.252393 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52334: BlockingRecv error: recv got EOF from 127.0.0.1:52334 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 23:01:22.263092 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.262109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.1.166.61:46883)
0620 23:01:22.262161 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.262163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.262478 (+ 315us) server_negotiation.cc:407] Connection header received
0620 23:01:22.262582 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.262584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.262596 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.262625 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.263025 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52346: BlockingRecv error: recv got EOF from 127.0.0.1:52346 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 23:01:22.281683 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.280876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.1.166.61:46883)
0620 23:01:22.280930 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.280932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.281075 (+ 143us) server_negotiation.cc:407] Connection header received
0620 23:01:22.281115 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.281116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.281129 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.281209 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.281615 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52352: BlockingRecv error: recv got EOF from 127.0.0.1:52352 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 23:01:22.315779 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.314972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.1.166.61:46883)
0620 23:01:22.315020 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.315022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.315191 (+ 169us) server_negotiation.cc:407] Connection header received
0620 23:01:22.315235 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.315237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.315250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.315319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.315714 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52362: BlockingRecv error: recv got EOF from 127.0.0.1:52362 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 23:01:22.381675 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.380946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.1.166.61:46883)
0620 23:01:22.381009 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.381011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.381104 (+ 93us) server_negotiation.cc:407] Connection header received
0620 23:01:22.381135 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.381136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.381147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.381202 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.381612 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52372: BlockingRecv error: recv got EOF from 127.0.0.1:52372 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 23:01:22.511911 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.511125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.1.166.61:46883)
0620 23:01:22.511199 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.511201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.511213 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:01:22.511251 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.511252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.511264 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.511355 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.511843 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52388: BlockingRecv error: recv got EOF from 127.0.0.1:52388 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 23:01:22.769964 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:22.769279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.1.166.61:46883)
0620 23:01:22.769343 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:22.769345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:22.769386 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:01:22.769473 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:22.769474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:22.769487 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:22.769533 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:22.769900 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52396: BlockingRecv error: recv got EOF from 127.0.0.1:52396 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:01:23.079187 24545 catalog_manager.cc:1259] Loaded cluster ID: 6b00cf91f41a43d1a3d00cfcaa3bef43
I20260620 23:01:23.079262 24545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a: loading cluster ID for follower catalog manager: success
I20260620 23:01:23.080299 24545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a: acquiring CA information for follower catalog manager: success
I20260620 23:01:23.080638 24545 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 758b97133be1428e8dffb01b70280a6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:01:23.135746 24559 catalog_manager.cc:1259] Loaded cluster ID: 6b00cf91f41a43d1a3d00cfcaa3bef43
I20260620 23:01:23.135877 24559 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb: loading cluster ID for follower catalog manager: success
I20260620 23:01:23.136782 24559 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb: acquiring CA information for follower catalog manager: success
I20260620 23:01:23.137108 24559 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35495892363c499cb25bb3a42d71faeb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:01:23.286012 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:23.283249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.1.166.61:46883)
0620 23:01:23.283365 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:23.283366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:23.283408 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:01:23.283434 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:23.283435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:23.283499 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:23.283567 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:23.283724 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.284234 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.285087 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.285311 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.285379 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:23.285582 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:23.285738 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:23.285800 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:23.285908 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:01:23.285930 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260620 23:01:23.290922 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:23.287933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.1.166.60:46805)
0620 23:01:23.288061 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:23.288063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:23.288071 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:23.288117 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:23.288118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:23.288164 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:23.288238 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:23.288369 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.288876 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.289639 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.289865 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.289925 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:23.290090 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:23.290308 (+ 218us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:23.290455 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:23.290737 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:01:23.290788 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260620 23:01:23.296984 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:23.293491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.1.166.60:46805)
0620 23:01:23.293534 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:23.293537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:23.293646 (+ 109us) server_negotiation.cc:407] Connection header received
0620 23:01:23.293683 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:23.293683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:23.293696 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:23.293750 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:23.293941 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.294850 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.296129 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.296401 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.296458 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:23.296669 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:23.296713 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:23.296756 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:23.296844 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:01:23.296879 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:23.296979 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:23.293261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.1.166.62:39991)
0620 23:01:23.293461 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:23.293464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:23.293794 (+ 330us) server_negotiation.cc:407] Connection header received
0620 23:01:23.293848 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:23.293849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:23.293907 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:23.294136 (+ 229us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:23.294234 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.295036 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.295953 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.296199 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.296277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:23.296565 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:23.296693 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:23.296760 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:23.296829 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 23:01:23.296858 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260620 23:01:23.297398 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:23.293263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.1.166.61:46883)
0620 23:01:23.293339 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:23.293342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:23.293608 (+ 266us) server_negotiation.cc:407] Connection header received
0620 23:01:23.293687 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:23.293688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:23.293707 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:23.293750 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:23.293947 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.294723 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.296513 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:23.296822 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:23.296887 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:23.297112 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:23.297153 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:23.297181 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:23.297295 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:01:23.297324 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:01:23.299793 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 24310
I20260620 23:01:23.305166 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 24385
I20260620 23:01:23.310267 1688 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskE9BVnS/build/release/bin/kudu with pid 24460
2026-06-20T23:01:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2860 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5657 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (307554 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS